An adaptive allocation and transmission method and system of audio and video resources

Abstract

The application provides an adaptive allocation and transmission method and system for audio and video resources, which comprises the following steps: parsing the audio and video fragments to be transmitted to obtain candidate resource segments and merging them to generate an audio and video resource unit set; generating a sacrificial mark for each audio and video resource unit according to resource types, attachment relationships and the like; in a current transmission period, converting the resource units into actual transmission tasks according to the sacrificial mark and transmission budget, and selecting an actual transmission quality version for the degradable resources; generating an experience debt record according to the actual processing mode and quality difference; and performing compensation transmission according to the compensation level and budget constraints in the subsequent compensable transmission period. The application realizes fine-grained scheduling of audio and video resources, preferentially guarantees key content, and converts the degradation or omission state into an experience debt for cross-period targeted compensation, thereby significantly improving the user experience in complex business scenarios.

Description

Technical Field

[0001] This invention belongs to the field of audio and video transmission and multimedia communication technology, and particularly relates to an adaptive allocation and transmission method and system for audio and video resources. Background Technology

[0002] With the development of services such as video-on-demand, live streaming, online conferencing, online education, cloud monitoring, in-vehicle entertainment, and multi-terminal collaborative playback, the objects processed by audio and video transmission systems have expanded from single video streams to various interrelated audio and video resources, including audio resources, text display resources, main moving images, static background images, local image quality data, and enhanced quality data attached to basic resources. Existing adaptive audio and video transmission is typically based on technologies such as segmented transmission, adaptive bitrate switching, layered encoding, and local enhancement layer transmission. The client or server switches between different bitrate versions or chooses whether to send a specific enhancement layer based on bandwidth, buffer, and sending window status. While this method can reduce stuttering and maintain continuous playback to some extent, in practical engineering, it still mainly uses complete segments, complete frame groups, complete bitrate layers, or fixed bitrate levels as scheduling objects, lacking a fine-grained organization method that can be directly used for transmission task allocation for different resources within the same segment. For example, in online education segments, audio explanations, whiteboard text, teacher images, background textures, and high-definition edge enhancement data of the whiteboard exist simultaneously within the same time window, but their contributions to the user's understanding of the course content are not the same. In sports live streaming segments, commentary audio, scoreboard text, main sports, audience background, and high-definition enhancement data of the scoreboard also have different transmission values. Existing methods often only reduce the overall bitrate, discard the enhancement layer, or roughly reduce the image quality when there is a weak network or insufficient transmission budget. This can easily lead to the compression of key text, key audio, main sports movements, and low-value backgrounds, resulting in continuous playback but difficulty in identifying key information. On the other hand, after a degradation, delay, or omission occurs in the current cycle, existing transmission systems usually only restore a higher bitrate in subsequent cycles, lacking a mechanism to record information such as "which resource unit was degraded, delayed, or omitted," "where the corresponding compensation data is located," and "whether the compensation still has timeliness value" as schedulable objects. This results in the system being unable to provide targeted compensation for previously substandard whiteboard text enhancement data, scoreboard high-definition data, subject motion supplementary data, or audio enhancement data after the network or transmission budget is restored. It remains at the level of subsequent segment quality improvement. The main shortcomings of existing technologies lie in the lack of continuity between resource organization granularity, sacrifice boundaries, actual transmission task records, and subsequent compensation scheduling. This makes it difficult to meet the engineering requirements of complex audio and video services, which prioritize the availability of critical content, allow for budget concessions on non-critical content, and compensate for critical quality losses in subsequent cycles. Summary of the Invention

[0003] This invention discloses an adaptive allocation and transmission method and system for audio and video resources to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, the first aspect of the present invention provides an adaptive allocation and transmission method for audio and video resources, the method comprising: The audio and video segments to be transmitted are parsed to obtain candidate resource segments. Candidate resource segments with the same resource type, continuous playback time, and screen area or dependency relationship that meet the merging conditions are merged to generate a set of audio and video resource units. Each audio and video resource unit includes resource unit number, segment number, time range, resource type, screen area range, target transmission quality, and dependency relationship. Based on the resource type, the dependency relationship, the target transmission quality, the time range, and the screen area range, a sacrificial tag is generated for each audio and video resource unit. The sacrificial tag includes a mandatory tag, a degradeable tag, a delayable tag, or an omissionable tag. Within the current transmission cycle, based on the sacrificiality flag and the current transmission cycle transmission budget, each audio / video resource unit is converted into a priority transmission task, a degraded transmission task, a delayed transmission task, or a currently omitted task, to obtain the actual transmission task set for the current cycle. For the audio and video resource units corresponding to the degradable marker, select the quality version that meets the budget constraint from the available quality versions as the actual transmission quality version; Based on the actual processing method, target transmission quality, and actual transmission quality of each audio and video resource unit in the current period's actual transmission task set, an experience debt record bound to the resource unit number is generated. In subsequent compensable transmission cycles, compensation is sent to select the experience debt records to be compensated from the experience debt records according to the compensation level, validity period, and cumulative budget constraints.

[0005] Furthermore, the merging of candidate resource segments that are of the same resource type, have continuous playback time, and whose screen regions or dependencies meet the merging conditions specifically includes: Determine whether the resource types of two candidate resource segments are the same; if they are not the same, retain them as different audio and video resource units. If the resource types are the same, the time interval is calculated based on the start and end timestamps of the two candidate resource segments. When the time interval does not exceed the preset time connection threshold, the playback time is considered to be continuous. For resource types with a defined screen area, determine whether the screen areas of two candidate resource segments overlap, are adjacent, or belong to the same preset screen grid group; for audio resource types, determine whether two candidate resource segments are located on the same audio track; for enhanced quality resource types, determine whether two candidate resource segments are attached to the same base resource or adjacent base resources. When the resource types are the same, the playback time is continuous, and the screen area or dependency relationship meets the merging conditions, the two candidate resource segments are merged into the same audio and video resource unit.

[0006] Furthermore, the step of generating a sacrificiality tag for each audio / video resource unit specifically includes: Based on the resource types, basic sacrificiality tags are generated, wherein audio resource types and text display resource types are mapped to mandatory tags, main motion resource types and local image quality resource types are mapped to degradeable tags, enhanced quality resource types are mapped to deferred tags, and static background resource types are mapped to omission tags. For audio and video resource units of the enhanced quality resource type that have a dependency relationship, the sacrificiality flag of the dependent basic audio and video resource unit is read and overwritten: when the sacrificiality flag of the dependent basic audio and video resource unit is a mandatory flag, the sacrificiality flag of the enhanced quality resource type audio and video resource unit is set to a deferred flag; when the sacrificiality flag of the dependent basic audio and video resource unit is an omissionable flag, the sacrificiality flag of the enhanced quality resource type audio and video resource unit is set to an omissionable flag.

[0007] Furthermore, the step of selecting the quality version that meets the budget constraint from the available quality versions as the actual transmission quality version specifically includes: Read the set of available quality versions of the audio and video resource units corresponding to the degradable marker, and sort the set of available quality versions from low to high according to transmission quality; Select a quality version that meets the budget constraint from the set of available quality versions as the actual transmission quality version. The budget constraint is that the sum of the data size corresponding to the selected quality version and the attached protection transmission amount does not exceed the remaining transmission budget of the current period. The attached protection transmission amount is the minimum compensation entry data amount of the latency enhancement resource that is attached to the mandatory resource and has a compensation priority not lower than the current degradeable resource in the current fragment.

[0008] Furthermore, the experience debt record includes debt number, resource unit number, fragment number, debt type, target compensation quality, compensation priority, validity period, compensation data index, and debt status; the debt type includes quality difference debt, delayed transmission debt, currently omitted debt, or partially transmitted debt; wherein: For audio and video resource units that are actually processed through downgraded transmission, quality difference debt is generated; For audio and video resource units that are actually processed by delayed transmission, a delayed transmission debt is generated; For audio and video resource units that are currently omitted in the actual processing method, generate the currently omitted debt; For audio and video resource units whose transmission status is partially transmitted, a partial transmission debt is generated.

[0009] Furthermore, in subsequent compensable transmission cycles, selecting the experience debt records to be compensated from the experience debt records and performing compensation transmission according to the compensation level, validity period, and cumulative budget constraint specifically includes: Filter the set of valid experience debt records from the experience debt records, which are in a debt status of pending or partial compensation, whose effective period has not yet expired, and whose compensation data index can be read. The set of valid experience debt records is sorted according to compensation level, validity period, and compensation data size; The system selects items one by one from the sorted results and adds them to the current period's compensable experience debt set until the cumulative compensation data size exceeds the compensation budget for the current compensation transmission period.

[0010] Furthermore, the compensation level is determined based on the resource type: Audio resource types, text display resource types, and enhanced quality resource types that are dependent on essential resources correspond to the experience debt records in the first compensation level; The experience debt records corresponding to the main motion resource type and the local image quality resource type belong to the second compensation level; The experience debt records corresponding to static background resource types and enhanced quality resource types attached to static background resources belong to the third compensation level.

[0011] Furthermore, the execution of compensation transmission specifically includes: Read the corresponding compensation data packet according to the compensation data index in the experience debt record and write it into the sending queue; After sending, the debt status of the experience debt record will be updated to compensated; When only a portion of the compensation data packet is sent, the debt status of the experience debt record is updated to partially compensated, and the sent boundary and the index of the remaining compensation data are recorded. When the experience debt record expires, the debt status of the experience debt record will be updated to waiver of compensation.

[0012] Furthermore, the method also includes: When the data packet of the audio / video resource unit corresponding to the mandatory protection mark exceeds the remaining transmission budget of the current period, it is divided and sent according to the encapsulation packet boundary, encoding packet boundary, or text display data boundary. The basic packet that can be decoded, played, or displayed by the receiving end is written first, and the incomplete transmission status of the audio / video resource unit is recorded in the actual transmission task set of the current period.

[0013] A second aspect of the present invention provides an adaptive allocation and transmission system for audio and video resources, the system comprising: The resource unit generation module is used to parse the audio and video segments to be transmitted to obtain candidate resource segments. It merges candidate resource segments with the same resource type, continuous playback time, and screen area or dependency relationship that meet the merging conditions to generate a set of audio and video resource units. Each audio and video resource unit includes resource unit number, segment number, time range, resource type, screen area range, target transmission quality, and dependency relationship. The sacrificiality marker generation module is used to generate sacrificiality markers for each audio and video resource unit based on the resource type, the dependency relationship, the target transmission quality, the time range, and the screen area range. The sacrificiality markers include mandatory markers, degradeable markers, delayed markers, or omissionable markers. The actual transmission task generation module is used to convert each audio and video resource unit into a priority transmission task, a degraded transmission task, a delayed transmission task, or a currently omitted task within the current transmission cycle, based on the sacrificiality flag and the current transmission cycle transmission budget, to obtain the set of actual transmission tasks for the current cycle. The quality version selection module is used to select, from the available quality versions, the quality version that meets the budget constraint as the actual transmission quality version for the audio and video resource unit corresponding to the degradable mark. The experience debt generation module is used to generate an experience debt record bound to the resource unit number based on the actual processing method, target transmission quality and actual transmission quality of each audio and video resource unit in the current period's actual transmission task set. The compensation transmission module is used to select the experience debt record to be compensated from the experience debt records and perform compensation transmission in subsequent compensateable transmission cycles, according to the compensation level, validity period and cumulative budget constraints.

[0014] The beneficial technical effects of the present invention are at least as follows: To address the aforementioned issues, this invention provides an adaptive allocation and transmission method and system for audio and video resources. It overcomes the limitations of traditional scheduling methods that rely on complete segments or fixed bitrates, refining audio, text, subject motion, and augmented data within a segment into independent resource units, achieving fine-grained scheduling. By generating sacrificial markers, the system can accurately distinguish resource value when the network is constrained, prioritizing critical audio, text, and subject motion, allowing low-value resources to relinquish budget within boundaries, and preventing overall bitrate reduction from making critical information difficult to identify. Simultaneously, the solution introduces an experience debt mechanism, converting degraded, delayed, or omitted states into traceable debt records, enabling targeted compensation based on compensation levels and budget constraints upon subsequent network recovery. This solution extends single-cycle bitrate adaptation to a cross-cycle resource allocation and compensation closed loop, significantly improving the user experience in complex business scenarios. Attached Figure Description

[0015] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0016] Figure 1 This is a flowchart of the adaptive allocation and transmission method for audio and video resources in an embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram of the candidate resource segment merging logic in an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of the sacrificiality marker generation logic in an embodiment of the present invention.

[0019] Figure 4 This is a schematic diagram illustrating the actual transmission task generation and quality version selection in an embodiment of the present invention.

[0020] Figure 5 This is a framework diagram of the adaptive allocation and transmission system for audio and video resources in an embodiment of the present invention.

[0021] Figure 6 This is a diagram showing the transmission relationship between five resource units in the same online education segment in this embodiment of the invention. Detailed Implementation

[0022] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0023] like Figure 1 As shown, the method includes steps S101 to S104.

[0024] like Figure 2 As shown, in step S101, candidate resource segments are obtained by parsing the audio and video segments to be transmitted. Candidate resource segments with the same resource type, continuous playback time, and screen area or dependency relationship that meet the merging conditions are merged to generate a set of audio and video resource units. Each audio and video resource unit includes resource unit number, segment number, time range, resource type, screen area range, target transmission quality, and dependency relationship.

[0025] Specifically, step S101 involves obtaining the audio and video segments to be transmitted at the sending end, server, or edge node. Execution follows. The system first parses... The parsing process uses segment indexes, encapsulation lists, audio tracks, video frame groups, subtitle or text overlay data, encoding level indexes, and enhancement layer indexes to obtain multiple candidate resource segments. A candidate resource segment is the smallest identifiable object before subsequent merging, and it must contain at least the candidate segment number, its corresponding segment number, start and end timestamps, resource type, screen area range, candidate quality version, data size index, and dependency relationship. Audio resource candidate segments come from audio packets or audio frame groups; text display resource candidate segments come from subtitle tracks, screen text recognition results, or business-side text display data; main motion resource candidate segments come from motion areas or main areas within video frame groups; static background resource candidate segments come from long-term stable areas with low business value; local image quality resource candidate segments come from whiteboard areas, scoreboard areas, alarm text areas, or screen shared text areas; enhanced quality resource candidate segments come from text edge enhancement layers, local sharpness enhancement layers, audio high-frequency supplementation layers, or supplementary quality data that can be resent. This parsing process enables... , Candidate resource segments have a clear origin, rather than appearing out of thin air in the merge formula.

[0026] After parsing the candidate resource segments, the system merges them. The merging is not based on a comprehensive scoring method, but rather on a hierarchical judgment according to more direct engineering constraints in audio and video transmission scheduling. For two candidate resource segments... and The system first determines whether the two resources belong to the same resource type, then determines whether they are consecutive in playback time, and finally determines whether they can be merged in terms of screen area or dependency relationship. This processing method is consistent with the logic of segment merging, subtitle segment merging, and screen area merging in actual audio and video engineering. That is, only when two candidate resource segments can be processed by the same transmission task in subsequent transmissions will they be merged into the same audio and video resource unit.

[0027] Specifically, the system first reads the candidate resource segments. and candidate resource segments The system retrieves the resource type field. If the resource types are different, they are directly retained as different audio and video resource units. For example, text display candidate resource segments and static background candidate resource segments are not merged even if they are in the same playback period because their subsequent transmission value, degradation method, and compensation method are different. If the resource types are the same, the system continues to read the start and end timestamps of the two and calculates the time interval between them. When the time interval does not exceed the preset time connection threshold, the two are considered to be continuous in playback time. For text display resources, main motion resources, static background resources, and local image quality resources, the system further reads the image area range of the two to determine whether the image ranges of the two overlap, are adjacent, or belong to the same preset image grid group; for audio resources, the system reads the audio track number and time continuity relationship of the two; for enhanced quality resources, the system reads the base resource number to which the two are attached to to determine whether the two are attached to the same base resource or adjacent base resources.

[0028] The above merging relationship can be expressed as: ; in, Indicates candidate resource segment and candidate resource segments Whether to merge them into the same audio and video resource unit, take the result if the condition is met, and take the result if the condition is not met; and All are composed of audio and video fragments to be transmitted. The candidate resource segments obtained through parsing; and Representing candidate resource segments respectively and candidate resource segments The resource type is derived from the track type and candidate resource segment parsing results, such as audio resources, text display resources, subject motion resources, static background resources, local image quality resources, or enhanced quality resources; Indicates candidate resource segments and candidate resource segments The time interval is calculated from the start and end timestamps of both. This indicates the preset time connection threshold, which is given by the transmission scheduling configuration of the server or edge node; This indicates whether the two resources meet the merging criteria in terms of spatial scope or dependency relationship. For resources with a defined image area, The determination is based on whether the two image areas overlap, are adjacent, or belong to the same preset image grid group; for audio resources... Determined by whether the two are on the same audio track; for enhanced quality resources... The determination is based on whether the two resources are attached to the same or adjacent basic resources. This decision formula is derived from the combination of Boolean conditions in engineering rules, and its function is to group candidate resource segments with the same resource type, continuous time, and mergeable spatial or dependency relationships into a single audio / video resource unit.

[0029] In this judgment formula, , , and All are Boolean judgment results and do not have physical dimensions; and The same time unit, either milliseconds or seconds, is used and maintained consistently within the same transmission task. If a candidate resource segment lacks a timestamp, resource type, or region / dependency field, the system does not perform automatic merging. Instead, it retains the candidate resource segment as a separate audio / video resource unit and marks it as a unit to be verified. This ensures that subsequent steps can still track the resource using a stable number, preventing unclear merging results due to missing fields.

[0030] For example, in an online education segment, candidate resource segments and candidate resource segments These are whiteboard text regions from adjacent video frames. The system reads the resource type fields of both and then determines... and Both are text display resources; after the system reads the start and end timestamps of both, it finds that the time interval between them does not exceed the preset time connection threshold. After reading the text display areas of both texts, the system determined that they overlapped or were adjacent to each other in the right-side area of ​​the screen. Established. At this time. Once established, the system will and Merge them into the same text display resource unit. Conversely, if the candidate resource segment Candidate resource segments for text display resources If it is a static background resource, then due to its relationship with... Even if two resources are adjacent in time or have similar screen areas, they will not be merged. This avoids merging resources with completely different subsequent processing methods into the same resource unit.

[0031] In sports live broadcast segments, if the scoreboard is located in the upper left corner of the screen for multiple consecutive frames and its display range is stable, the system will merge these text display candidate resource segments into one text display resource unit. If the main moving area continuously moves in adjacent frames, but its covering grid belongs to the same preset motion area group, the system will merge these main motion candidate resource segments into one main motion resource unit. If the audience background is consistently at the top of the screen and changes little, the system will merge adjacent static background candidate resource segments into one static background resource unit. For enhanced quality resources, the system will not merge solely based on temporal proximity, but will first check whether the underlying resources to which the enhanced quality resources are attached are consistent. For example, if two subtitle edge enhancement candidate segments are both attached to the same text display resource, they can be merged; if one enhancement candidate segment is attached to a text display resource and the other is attached to a local image quality resource, they will remain as different enhanced quality resource units.

[0032] After merging, the system generates a unified description for each audio / video resource unit. This description includes the resource unit number, fragment number, time range, resource type, screen area range, target transmission quality, and dependency relationship. The resource unit number is generated from the fragment number and the resource sequence number. For example, when the fragment number is seg_102, the third resource unit can be recorded as seg_102_u003; the fragment number comes from the input fragment. The playlist or transmission list; the time range comes from the start and end timestamps of audio frames, video frames, or text display data; the resource type comes from the track type and candidate resource segment parsing results; the screen area range comes from the subtitle display area, video block index, or fixed grid division results; the target transmission quality comes from the target bitrate version corresponding to the current segment, the enhancement layer configuration, the quality level in the server list, or the original encoding quality marker; the dependency relationship is used to record the binding relationship between the enhanced quality resources and the basic resources. After the above processing, this step outputs a set of audio and video resource units. . Each audio / video resource unit in the system has a stable resource unit number, which will continue to be used in subsequent processes such as sacrificiality tag generation, actual transmission task generation, and experience debt generation.

[0033] like Figure 3 As shown, in step S102, a sacrificial marker is generated for each audio and video resource unit based on the resource type, dependency relationship, target transmission quality, time range, and screen area range. The sacrificial marker includes a mandatory marker, a degradeable marker, a delayable marker, or an omissionable marker.

[0034] Specifically, this step receives the audio and video resource unit set output in step S101. and to Each audio / video resource unit generates a sacrificiality tag, resulting in a set of sacrificiality tags. Step S101 has already established stable resource unit numbers, segment numbers, time ranges, resource types, screen area ranges, target transmission quality, and dependencies for each audio / video resource unit. This step directly reads these fields and converts them into processing boundaries that subsequent transmission tasks can execute. Sacrificeability markers include mandatory protection markers, degradeable markers, delayable markers, and omissionable markers. Mandatory protection markers correspond to subsequent priority transmission tasks, degradeable markers correspond to subsequent degraded transmission tasks, delayable markers correspond to subsequent delayed transmission tasks, and omissionable markers correspond to omissions within the current playback window. This processing further transforms the audio / video resource units generated in step S101 from "internal resource objects within segments" into "resource objects with clearly defined transmission boundaries," providing a direct basis for step S103 to generate actual transmission tasks according to the current transmission budget.

[0035] The input for this step is a set of audio and video resource units. Any audio or video resource unit is denoted as The system reads sequentially. The resource unit number, segment number, time range, resource type, screen area range, target transmission quality, and dependency relationship are defined. The resource unit number ensures that the sacrificiality marker generated in this step is bound to the same audio / video resource unit as the subsequent actual transmission task and experience debt; the segment number identifies the audio / video segment to which the resource unit belongs; the time range determines the persistence of the resource unit within the current playback window; the resource type distinguishes between audio resources, text display resources, subject motion resources, static background resources, local picture quality resources, and enhanced quality resources; the screen area range determines the position and coverage of text display, subject motion, or static background in the screen; the target transmission quality determines whether there is an executable degradation level for the resource unit; and the dependency relationship determines the binding relationship between the enhanced quality resource and the basic audio resource, text display resource, or local picture quality resource.

[0036] The system first reads the pre-configured sacrificiality rule template. This rule template can be stored in the transmission scheduling configuration file on the server, edge node, or sender, or it can be stored as a database rule table, protocol configuration field, or business scenario configuration item. The rule template includes basic resource type rules, dependency relationship coverage rules, quality level correction rules, and playback window correction rules. The basic resource type rules are used to... The resource types are given initial sacrificiality tags. For example, audio resources and text display resources are mapped to mandatory tags, main motion resources and local image quality resources are mapped to degradeable tags, enhanced quality resources are mapped to deferred tags, and static background resources are mapped to omissionable tags. This mapping originates from the production rules in the engineering rule system, and its basic form is "if the condition is met, execute the corresponding action," which is a common expression in computer rule engines and Boolean logic. Based on this, this application limits the rule conditions to the audio and video resource unit fields formed in step S101, and limits the rule actions to four sacrificial tags that can directly drive the transmission task, so that the tagging results can directly enter the subsequent transmission scheduling.

[0037] Basic tag generation can be represented as: ; in, Represents audio and video resource units The basic sacrificiality marker; Represents a set of audio and video resource units One of the audio and video resource units; express The resource type field is derived from the resource unit description generated in step S101; This represents the mapping rule from "resource type to base tag" in the sacrificiality rule template, specifically provided by the transmission scheduling configuration of the server, edge node, or sender. For example, When it is an audio resource, Output must-keep flag; When the main sports resource is used, Output a degradeable flag; To enhance quality resources, The output can be delayed. Both sides of the formula are the results of tag enumeration. The right side obtains the basic tag on the left side through rule mapping. The calculation objects belong to the same type of discrete tags.

[0038] Get the basic marker The system first generates sacrificial tags for basic audio and video resource units without dependency relationships, and then applies dependency coverage to enhanced quality resources with dependency relationships. Basic audio and video resource units include audio resources, text display resources, subject motion resources, static background resources, and local image quality resources; enhanced quality resources include text edge enhancement resources, high-frequency audio supplementation resources, and local image sharpness enhancement resources. The system reads... The dependency relationship field, if To enhance the quality of resources and attach them to a basic audio and video resource unit, then in The system searches for the resource unit number of the basic audio / video resource unit and reads the sacrificial flag already generated for that basic audio / video resource unit. For enhanced quality resources attached to mandatory resources, the system sets their flag to a deferred flag, allowing the enhanced quality resource to relinquish its budget when the current transmission budget is insufficient, while retaining an entry point for subsequent compensation transmission. For enhanced quality resources attached to optional resources, the system sets their flag to an optional flag. For enhanced quality resources attached to degradeable resources, the system sets them to a deferred or degradeable flag based on the business configuration in the rule template. For example, in online education scenarios, whiteboard text display resources are mandatory resources, and their high-definition edge enhancement resources are set to a deferred flag; in sports live streaming scenarios, scoreboard text display resources are mandatory resources, and their high-definition enhancement resources are set to a deferred flag; static background texture enhancement resources attached to static background resources are set to an optional flag.

[0039] Dependency coverage can be represented as: ; in, This indicates a salvageable marker after dependency relationships have been covered. This represents the basic sacrificiality marker obtained from the resource type basic rules; express The dependency relationship field comes from the resource unit description generated in step S101; This indicates that the audio / video resource unit has an attached object; express The underlying audio and video resource units on which it is attached are numbered by Specify, and can be used in the audio and video resource unit set Found in the middle; Represents basic audio and video resource units The sacrificiality flag is generated in this step before the attached quality enhancement resource. This formula originates from a Boolean conditional branching rule, and its derivation is as follows: first, determine if the quality enhancement resource has an underlying dependent object; then, read the flag of the underlying resource; finally, constrain the flag of the quality enhancement resource to a processing boundary that aligns with the transmission value of the underlying resource. Both sides of this formula are flag enumeration results, the condition is a Boolean judgment result, and the branch output is the sacrificiality flag.

[0040] After the dependency relationship is overridden, the system reads... The target transmission quality and available quality level, for Adjust the quality level. If If the current target transmission quality has an executable next-level quality level, for example, if a local image quality resource has three quality levels (high, medium, and low), or if the main motion resource has a base layer and an enhancement layer, then the downgrade marker has an actual execution basis; if For resources with only a single base quality level, such as a single subtitle layer, base audio layer, or base keyframe data, the system maintains it as either mandatory or optional based on the resource type. For enhanced quality resources, where the target transmission quality typically corresponds to supplementary data rather than base playback data, the system prioritizes maintaining the deferred flag. Subsequently, the system reads... Based on the time range and screen area range, the system performs playback window correction on key display resources within the current playback window. For text display resources, if their time range covers most of the playback window of the current segment, or their screen area range is located in the subtitle area, whiteboard area, score area, screen-shared text area, or alarm text area in the service configuration, then the mandatory protection mark is maintained. For main motion resources, if their time range covers multiple consecutive frames and corresponds to the main activity area in the screen, then the degradeable mark is maintained. If the main motion resource is a base layer reference resource, the rule template can promote it to the mandatory protection mark. For static background resources, the system maintains the omissionable mark or the degradeable mark according to the resource type, dependency relationship, and service configuration. The result after quality level correction and playback window correction is denoted as... And write it into the sacrificiality tag set. .

[0041] In the process of generating tags for online education segments, assuming that step S101 outputs... It contains five audio and video resource units: teacher-explained audio resources. Blackboard text display resources Teacher's main subject movement resources High-definition edge-enhanced resources for blackboard text Background wall texture enhancement resources System read resource types For audio resources, obtained according to basic rules The tag is set to "Guaranteed", the dependency relationship is empty, the target transmission quality is the basic audio quality, and the final tag is "Guaranteed". The system reads... The resource type is text display resource, and it is marked as mandatory according to the basic rules. The time range covers segments from 0.3 seconds to 1.9 seconds, the screen area is located within the whiteboard area, and the playback window remains marked as mandatory even after correction. The system reads... The resource type is the main motion resource. Based on the basic rules, it is marked as degradable, and its target transmission quality has multiple levels, ultimately maintaining the degradable mark. The system reads... The resource type is enhanced quality resource, the basic rules obtain a deferable tag, and the dependency field points to... ,because The marker is a mandatory marker, and according to the dependency relationship overriding rule, it is a mandatory marker. Maintain a deferred flag. System read. The resource type is an enhanced quality resource, the dependency field points to a static background resource, and the tag for the static background resource is optional. The marker is set to be optional. This processing result causes the audio and whiteboard text to be prioritized for transmission in subsequent step S103, the main motion to be degraded for transmission, the high-definition edge enhancement resources of the whiteboard to be delayed for transmission, and the background texture enhancement resources to be omitted in the current playback window.

[0042] In the process of tagging and generating sports live broadcast segments, if The system includes commentary audio resources, scoreboard text display resources, sports subject resources, static background resources of the stands, and high-definition enhanced scoreboard resources. All resources are processed according to the same rules. Commentary audio resources are marked as "must-protect" based on the basic rules for audio resources; scoreboard text display resources are marked as "must-protect" based on the basic rules for text display resources, and since their display area is within the score display area, they retain this "must-protect" mark. Sports subject resources are marked as "degradeable" based on the basic rules for sports subject resources; static background resources of the stands are marked as "omittable" based on the basic rules for static background resources; and high-definition enhanced scoreboard resources are marked as "delayable" based on the basic rules for enhanced quality resources, and since their attachment is the scoreboard text display resource, their basic resource mark is "must-protect," thus retaining the "delayable" mark. This results in the following behavior during subsequent transmission: commentary audio and scoreboard text prioritize the transmission budget; sports subject data is allowed to maintain continuity at a lower quality; high-definition enhanced scoreboard data enters the subsequent compensation entry point; and the static background of the stands relinquishes the transmission budget within the current window.

[0043] This step outputs a set of sacrificiality markers. . Each item in is related to Each audio / video resource unit corresponds one-to-one with the others, and each item contains at least a resource unit number, a fragment number, and a sacrificiality flag. The resource unit number and fragment number are taken from the output of step S101. The sacrificiality marker is generated in this step based on resource type, dependency relationship, target transmission quality, time range, and screen area range. will with Proceed to step S103, which generates the actual transmission task set for the current period. Step S103 reads... Then it can be determined which audio and video resource units enter the priority transmission task, which enter the degraded transmission task, which enter the delayed transmission task, and which are omitted from the current playback window.

[0044] like Figure 4 As shown, in step S103, within the current transmission cycle, each audio / video resource unit is converted into a priority transmission task, a degraded transmission task, a delayed transmission task, or a currently omitted task based on the sacrificiality flag and the current transmission cycle transmission budget, thus obtaining the actual transmission task set for the current cycle; for the audio / video resource unit corresponding to the degradeable flag, the quality version that meets the budget constraint is selected from the available quality versions as the actual transmission quality version.

[0045] Specifically, this step receives the set of sacrificiality markers output from step S102. and based on The resource unit number is used to read the description field of the corresponding audio / video resource unit and generate the actual transmission task set for the current period. Simultaneously, the data packets that can be sent in the current period are written into the sending queue. Step S101 has already organized the audio and video fragments to be transmitted into a set of audio and video resource units. Step S102 has already... Each audio / video resource unit is converted into a mandatory marker, a degradeable marker, a delayed marker, or an omissionable marker; this step follows up on this result, converting the markers into specific transmission actions within the current transmission cycle. The system reads... Each item in the resource unit, fragment number, and sacrificiality flag is used to obtain the time range, resource type, screen area range, target transmission quality, and dependency relationship of the corresponding audio / video resource unit in step S101 through the resource unit number. The current cycle transmission budget is denoted as... This is given by the sender or edge node at the start of the current transmission cycle, and its source can be the amount of data that the send queue allows to write, the amount of data that the protocol send window allows to send, or the amount of data allocated by the server for the current fragment. The system uses... Constrain the sending tasks for this period, ensuring that essential resources are prioritized for sending, degradeable resources are selected within the budget based on actual quality, deferred resources are reserved for processing in subsequent periods, and omitable resources are omitted in the current playback window.

[0046] In this step , , , , and All data volume units used are the same type, such as bytes, kilobytes, or the number of transmission packets, and are uniformly converted within the same task. If the system provides the budget in terms of protocol transmission windows, the transmittable amount of the window is first converted into bytes or packets before comparison; if the system provides the budget in terms of bitrate and period duration, the transmittable amount for that period is first obtained by multiplying the bitrate by the period duration. This ensures that the capacity constraints in steps S103 and S104 are performed under the same unit, avoiding the problem of mixing bitrate, duration, and data packet quantity.

[0047] The system first based on Establish a candidate task list for the current period. For audio and video resource units corresponding to the "must-protect" mark, the system generates priority transmission candidate tasks and reads the data packet index and data size corresponding to the target transmission quality of the resource unit. For audio and video resource units corresponding to the "degradeable" mark, the system reads its available quality version set, which comes from the bitstream list, coding level index, local quality level, or enhancement layer index corresponding to the target transmission quality in step S101. For audio and video resource units corresponding to the "delayable" mark, the system generates delay candidate tasks and records the underlying resource number, target transmission quality, and data index to be sent for the resource unit. For audio and video resource units corresponding to the "omitable" mark, the system generates current omission candidate tasks and records their resource unit number, target transmission quality, and omission status. In online education scenarios, audio explanations and whiteboard text typically form priority transmission candidate tasks, while the teacher's main motion in the video becomes a degraded candidate task, high-definition edge enhancement of the whiteboard text becomes a delayed candidate task, and enhanced background texture becomes a currently omitted candidate task. In sports live streaming scenarios, audio commentary and scoreboard text form priority transmission candidate tasks, the main subject of the movement becomes a degraded candidate task, high-definition enhancement of the scoreboard becomes a delayed candidate task, and the static background of the audience becomes a currently omitted candidate task.

[0048] During the priority transmission candidate task processing phase, the system establishes a transmission order based on playback time range and dependencies. For essential resources within the same playback window, the system first sends resources that form the basic playback continuity, and then sends resources that depend on that basic resource for interpretation. For example, in online conferencing and online education scenarios, audio resources enter the transmission queue first, followed by whiteboard text, subtitles, or screen text data; in sports live streaming scenarios, commentary audio and scoreboard text are at the same priority level, and both enter the transmission queue before the audience background and enhanced quality data. If the data packet of a certain essential resource exceeds the remaining budget for the current period, the system segments and sends it according to the encapsulation packet boundaries, encoding packet boundaries, or text display data boundaries, prioritizing the writing of basic packets that can be decoded, played, or displayed by the receiving end, and including them in the actual transmission task set for the current period. The system records the incomplete transmission status of the resource unit. This status retains the resource unit number, target transmission quality, and actual transmission quality, enabling subsequent steps to determine whether the resource unit has incurred experience debt.

[0049] During the degradable candidate task processing phase, the system selects the actual quality version that can be sent within the current budget for each degradable resource unit. The initial source of this selection rule is the capacity constraint feasibility judgment in communication scheduling and streaming media bitrate selection. Its basic idea is that the amount of data to be sent by the task should fall within the current remaining sending capacity. This application modifies this rule for audio and video resource units: on the one hand, the candidate object is no longer the entire bitstream, but the audio and video resource unit formed in step S101; on the other hand, a dependency protection quantity is introduced when selecting the quality version of degradable resources, ensuring that the enhanced resources dependent on the guaranteed resources do not completely lose their subsequent compensation entry point because a certain degradable screen resource fills the budget in the current cycle. The actual quality version selection for the current cycle is expressed as: ; in, Represents audio and video resource units The actual transmission quality version selected in the current period; Indicates by The audio and video resource unit pointed to by the resource unit number in the middle has its time range, resource type, target transmission quality and dependency relationship derived from the resource unit description formed in step S101; express The available quality versions are derived from the fragment bitstream list, coding level index, local quality level, or enhancement layer index, such as high-quality, medium-quality, and basic quality versions. Pre-sorted according to transmission quality from low to high This indicates that the highest quality version is selected from the quality versions that meet the budget constraints. express One of the available quality versions; express In quality version The size of the data to be sent is determined by the fragmentation index, encapsulation manifest, or encoding level index. Indicates the relationship with the current resource unit The amount of attached protection sent in relation to the fragment and resource level, when there are resources attached to the mandatory protected resources in the fragment and the compensation priority is higher than or equal to the specified priority. When delaying resource enhancement, The rule template is provided at the resource level to reserve the minimum compensation entry point for this type of latency enhancement resource; when no such latency enhancement resource exists... Pick ; This represents the remaining transmission budget for the current period after deducting scheduled priority transmission tasks, and is determined by the current period's transmission budget. The formula is derived from the feasibility assessment of capacity constraints, decreasing with task scheduling. It first lists the set of available quality versions for a resource unit, then reads the data size corresponding to each quality version, subsequently adds the dependent protection transmission volume related to the fragment and resource level of the current resource unit, and finally selects the highest quality version that satisfies the remaining budget constraint. (The constraint conditions are...) , and All are objects capable of sending data; the comparison is performed between data objects of the same type; left side With the set on the right All of these represent the quality version selection results.

[0050] To avoid having too low a budget It cannot be determined; the system checks if the set of feasible quality versions is empty before executing the expression. If at least one quality version satisfies... If no feasible version exists, the system selects the highest quality version according to the above formula; if no feasible version exists, but the resource unit has a base package that can be independently decoded or displayed, the system prioritizes writing the base package and... The system records part of the transmission status; if the base packet cannot be written to the current period either, the system does not generate a normal degraded transmission task, but instead... The resource unit is recorded as a delayed non-transmission state and processed as a delayed transmission debt in step S104. Through this boundary processing, , The process maintains continuity with subsequent debt generation, preventing situations where degradeable resources lack actual quality versions and cannot proceed to subsequent steps.

[0051] Taking online education segmentation as an example, step S102 marks the teacher's main motion resource as degradable, and step S101 records that the resource has high-quality, medium-quality, and basic-quality versions. The system reads the set of available quality versions. The results showed that: high-quality version corresponds to a transmission volume of 180, medium-quality version corresponds to a transmission volume of 110, and basic-quality version corresponds to a transmission volume of 70. Current cycle transmission budget. With a budget of 500, and 340 already occupied by the essential resources of audio explanation and whiteboard text, the remaining transmission budget... The value is 160. Because the whiteboard text also has a delayed high-definition edge enhancement resource, the rule template sets the dependency protection value for this period. The value is 40. Substituting the above formula into the system, a high-quality version requires... ,Exceed Medium quality version requires No more than The basic quality version requires... And not more than .therefore Select the medium-quality version. Based on this, the system generates a downgraded transmission task for the main motion resource, actually sends the data packets corresponding to the medium-quality version, and records in the task record that the target transmission quality is the high-quality version and the actual transmission quality is the medium-quality version. This calculation process ensures that the downgrade action when the budget is insufficient is based on a clear data version selection, rather than simply providing an abstract "reduce quality" instruction.

[0052] During the deferred candidate task processing phase, the system determines the recording method for deferred tasks based on dependency relationships. If the underlying resource to which the deferred resource depends has already been prioritized or downgraded for transmission in the current period, a deferred transmission task is generated for that deferred resource. This task includes the resource unit number, the dependent underlying resource number, the target transmission quality, and the index of the data to be transmitted. If the underlying resource is in a partially transmitted state in the current period, the deferred resource still enters the deferred transmission task, and the partial transmission state of the underlying resource is recorded. For example, in an online education scenario, high-definition edge enhancement resources for whiteboard text depend on whiteboard text display resources. If the whiteboard text display resources have already been transmitted, the high-definition edge enhancement resources enter the deferred transmission task, subsequently creating experience debt related to text clarity. In a sports live streaming scenario, high-definition enhancement resources for scoreboards depend on scoreboard text display resources. If the scoreboard text display resources have already been transmitted, the high-definition enhancement resources enter the deferred transmission task, subsequently creating experience debt related to scoreboard clarity. This process ensures that the dependency relationship between enhancement quality resources and underlying resources is preserved in the actual transmission task.

[0053] During the current omission candidate task processing phase, the system writes omissionable resources into the current cycle's actual transmission task set and records the current omission status. Omissionable resources typically include static background resources, texture enhancement resources attached to static backgrounds, and low-value visual decoration data within the current playback window. These types of resources are not written to the transmission queue in the current cycle, but their resource unit number, target transmission quality, and actual processing method are retained. For example, in live sports broadcasts, after the static background resources of the audience seats are currently omitted, the budget is concentrated on the commentary audio, scoreboard text, and the main sports content; in online education, after the background wall texture enhancement resources are currently omitted, the budget is concentrated on the explanation audio, whiteboard text, and the teacher's main visual movement. The current omission task enters... Subsequently, the next steps can determine whether the omission constitutes an experience debt that needs to be repaid, based on the resource type, validity period, and compensation value.

[0054] The actual processing method for the current cycle is determined by the sacrificiality flag output in step S102 and the actual quality version obtained in this step, and can be expressed as: ; in, Represents audio and video resource units The actual handling method in the current cycle; This indicates that step S102 is... The generated sacrificiality tags come from the set of sacrificiality tags. ; This indicates the actual transmission quality version selected based on the remaining budget for the current period; express The quality version corresponding to the target transmission quality comes from the target transmission quality field of the resource unit in step S101. When the actual transmission quality version of the degradable resource is lower than the target transmission quality version, the actual processing method is degraded transmission; when the degradable resource is sent according to the target transmission quality within the current budget, the actual processing method is target transmission. The delay flag and the omission flag form delayed transmission and current omission respectively. This rule is derived from Boolean conditional branching and finite state task assignment. Its derivation relationship is: first read the sacrificiality flag, then combine it with budget feasibility to obtain the actual quality version, and finally map the resource unit to the current cycle task state. In the formula, the output of each branch is the transmission task state, and the condition terms come from... Results of quality version selection.

[0055] formula With formula There is a sequential relationship. The system passes through first. For degradeable resources, select the quality version within budget, and then... The final task state is determined by combining the sacrificiality flag. For essential resources, deferred resources, and omissionable resources, Directly by The value is determined; for degradeable resources, Further comparison is needed. With target quality version For example, in the aforementioned online education segment, the target quality version of the teacher's main motion resource is a high-quality version, but... The medium quality version was selected, therefore For downgraded transmission; explaining audio resources To ensure the mark is preserved, therefore Prioritize transmission; write the high-definition edge enhancement resources on the whiteboard. It is a deferred tag, therefore For delayed transmission; background wall texture enhancement resources Since the marker can be omitted, therefore This is currently omitted. The processing result ensures that the actual transmission task set in the current period includes both the actually sent data packets and the delay and omission states, facilitating the formation of experience debt in subsequent steps.

[0056] In a complete online education transmission cycle, if The system includes five resource unit markers: audio explanation is mandatory, whiteboard text is mandatory, teacher's main motion is degradeable, high-definition edge enhancement of whiteboard text is deferred, and background texture enhancement is optional. The system first writes the audio explanation and whiteboard text into the priority sending queue; then, based on the remaining budget, it selects a medium-quality version for the teacher's main motion and generates a degraded transmission task; high-definition edge enhancement of whiteboard text enters the deferred transmission task, preserving its basic relationship with the whiteboard text; and background texture enhancement enters the currently omitted task. The current cycle's actual transmission task set. In the process, the audio and whiteboard text are sent as "sent," the teacher's main motion image is sent as "degraded," the high-definition edge enhancement of the whiteboard text is sent as "delayed and not sent," and the background texture enhancement is sent as "currently omitted." For the receiving end, this cycle acquires the audio, whiteboard text, and medium-quality main motion image. For subsequent steps, the delayed state of the high-definition edge enhancement of the whiteboard text and the degraded state of the teacher's main motion image can be identified as objects that may require compensation.

[0057] During a live sports broadcast, if the commentary audio and scoreboard text are marked as mandatory, the main sports content is marked as degradeable, the high-definition enhancement of the scoreboard is marked as delayed, and the static background of the audience is marked as optional, the system prioritizes sending the commentary audio and scoreboard text to ensure users continuously receive commentary and score information. When the budget is insufficient, the main sports content is sent to the lowest available quality version to maintain the continuity of the game. The high-definition enhancement of the scoreboard is sent in a delayed transmission task, with high-definition edge or resolution data to be sent later. The static background of the audience is currently omitted, allowing the budget to be concentrated on critical resources. This processing corresponds to the differentiated transmission actions of different audio and video resources within the same segment. The current cycle no longer generates only a single overall bitrate selection result, but rather forms a set of traceable actual transmission tasks.

[0058] This step outputs the set of actual transmission tasks for the current period. . Each item corresponds to an audio / video resource unit and includes at least the resource unit number, fragment number, actual processing method, target transmission quality, actual transmission quality, transmission status, and transmission data index. The resource unit number and fragment number are derived from... The resource unit description and its binding; the actual processing method is determined by this step based on the sacrificiality flag, budget feasibility and dependency relationship; the target transmission quality comes from the resource unit description formed in step S101; the actual transmission quality is determined by the quality version or transmission status selected in this step; the transmission status records whether the resource unit has been transmitted, partially transmitted, downgraded, delayed and not transmitted or currently omitted in the current period; the transmission data index is used to point to the data packet that has been written into the transmission queue or the data object to be processed later. This will be used as input to step S104, enabling step S104 to generate experience debt based on the actual processing method and actual transmission quality, and further arrange compensation transmission.

[0059] Step S104: Based on the actual processing method, target transmission quality, and actual transmission quality of each audio / video resource unit in the current cycle's actual transmission task set, generate an experience debt record bound to the resource unit number; in subsequent compensable transmission cycles, select the experience debt record to be compensated from the experience debt records and perform compensation transmission according to the compensation level, validity period, and cumulative budget constraints.

[0060] Specifically, this step receives the set of actual transmission tasks for the current period output by step S103. and based on The actual processing method, target transmission quality, actual transmission quality, transmission status, and transmission data index are recorded to generate an experience debt set. Subsequently, in subsequent compensable transfer cycles, experience debts that still have compensable value are selected and compensable transfers are performed to obtain the compensable transfer result. Step S103 has already converted the sacrificiality markers in step S102 into the actual transmission results of the current cycle. For example, an audio resource has been prioritized for transmission, a text display resource has been transmitted at the target quality, a main motion resource has been downgraded for transmission, an enhanced quality resource has been delayed for transmission, and a static background resource has been omitted. This step takes these actual transmission results and converts the downgraded, delayed, omitted, and partially transmitted states into compensation objects that can be processed in subsequent cycles. In this way, when the transmission budget allows, the system does not simply improve the overall quality of subsequent segments, but can go back to the specific resource unit number to determine which audio / video resource unit did not reach the target transmission quality, what data needs to be retransmitted, whether the compensation is still within the validity period, and how to update the status after the compensation is completed.

[0061] The input for this step is the actual set of transmission tasks for the current period. . Each item in the table corresponds to an audio / video resource unit and includes at least the resource unit number, segment number, actual processing method, target transmission quality, actual transmission quality, transmission status, and transmission data index. The resource unit number is used to bind the experience debt to the same audio / video resource unit formed in step S101; the segment number is used to determine the audio / video segment to which the debt belongs; the actual processing method is used to determine whether the resource unit is prioritized, targeted, downgraded, delayed, or currently omitted in step S103; the target transmission quality is used to determine the quality version originally expected for the resource unit; the actual transmission quality is used to determine the quality version actually achieved by the resource unit in the current period; the transmission status is used to distinguish between already transmitted, partially transmitted, downgraded transmitted, delayed not transmitted, and currently omitted; and the transmission data index is used to locate the supplementary data packets, enhancement layer data, local area data, or untransmitted data ranges that need to be read during subsequent compensation transmissions. At the start of a subsequent compensable transmission period, the system also reads the transmission budget for that period. The source of this budget is the same as in step S103, and can be given by the writable amount of the sender queue, the protocol sending window, or the quota allocated by the server. In this cycle, the system first ensures the sending of ordinary essential resources, and then allocates compensation budget from the remaining part to repay the experience debt.

[0062] The system first iterates through the set of actual transmission tasks for the current period. The system determines the actual processing method and transmission status of each audio / video resource unit. For resource units whose actual processing method is priority transmission and whose transmission status is "already transmitted," if the actual transmission quality reaches the target transmission quality, the resource unit becomes a completed state in the current period. For resource units whose actual processing method is target transmission and whose actual transmission quality equals the target transmission quality, they also become completed states. For resource units whose actual processing method is degraded transmission, the system generates quality difference debt. For resource units whose actual processing method is delayed transmission, the system generates delayed transmission debt. For resource units whose actual processing method is currently omitted, the system generates currently omitted debt. For resource units that have entered priority transmission, target transmission, or degraded transmission but whose transmission status is "partially transmitted," the system generates partial transmission debt. The experience debt record includes debt number, resource unit number, fragment number, debt type, target compensation quality, compensation priority, validity period, compensation data index, and debt status. The debt number can be obtained by concatenating the fragment number, resource unit number, and current period sequence number; the debt type is determined by the actual processing method and transmission status; the target compensation quality comes from the difference between the target transmission quality and the actual transmission quality; the compensation data index is obtained by... The data sent in the process and the dependency relationship formed in step S101 are jointly determined; the debt status is initially set to pending compensation.

[0063] The generation of experience debt records employs state transition rules from a finite state machine. The basic idea of ​​a finite state machine is to derive the next state or output record based on the current state and triggering conditions. This application applies these rules to the processing of audio and video resource transmission results, converting the actual processing method and sending state formed in step S103 into a compensable experience debt record. Its expression is as follows:

[0064] in, Represents audio and video resource units The type of experience debt corresponding to the current cycle; Represents the set of actual transmission tasks in the current period. The audio / video resource unit corresponding to a specific item in the document; This indicates the actual processing method for the current cycle determined in step S103, which comes from... ; express The sending status, from ; This indicates that the resource unit will not generate experience debt in the current period. Partial transmission debt is not limited to guaranteed resources, but also covers resource units that have entered the target transmission or degraded transmission but have only transmitted a portion of the data packets due to insufficient transmission window. The derivation process of this formula is as follows: first read... The actual processing method involves reading the sending status, mapping it to the state branches in the finite state machine, and finally outputting the debt type or an empty result. If... The system is based on The resource unit number, fragment number, target compensation quality, compensation priority, validity period, compensation data index, and debt status are used to generate a complete experience debt record, which is recorded as follows: and will Write experience debt set ;like The system will not generate an experience debt record for this resource unit. Therefore, Indicate the type of debt. This indicates a complete debt record, and the two should be kept separate in subsequent processing.

[0065] System generation Next, the compensation content for experience debt is determined. For quality difference debt, the system reads the difference between the target transmission quality and the actual transmission quality, and locates the data to be resent based on the transmitted data index. For example, if the target transmission quality of a certain subject motion resource is a high-quality version in step S103, but the actual transmission quality is a medium-quality version, then the compensation data index points to the local quality supplementary data or enhancement layer data between the medium-quality version and the high-quality version; if there is no differential supplementary data in the encoding link, then the compensation data index points to the corresponding local area data in the high-quality version that has not yet been sent. For delayed transmission debt, the system directly writes the data index to be sent recorded in step S103 into the debt record. For example, when the high-definition edge enhancement resource for whiteboard text is delayed, the compensation data index points to the high-definition edge enhancement data. For currently omitted debt, the system reads the resource type and validity period; if the resource is a static background texture enhancement resource, a shorter validity period and a lower compensation level are set; if the resource is a text display resource, audio resource, subject motion resource, or an enhancement quality resource attached to a mandatory resource, a higher compensation level is set. For partial transmission debt, the system reads the boundaries of the transmitted data packets and the boundary of the target data packets, and uses the remaining untransmitted basic packets or enhancement packets as the index for compensation data.

[0066] Subsequently, the system will adjust the experience debt set in subsequent compensable transmission cycles. Filtering and sending are performed. The compensable transmission period can be the next segment transmission period or the transmission period during which the current playback buffer allows for the insertion of supplementary data. The system reads the transmission budget for this period. First, arrange the transmission of essential resources, and then obtain the budget allocated to compensation transmission. The system from The system filters experience debts that are pending or partially compensated, have not yet expired, and whose compensation data index is readable, and organizes them into compensation queues according to compensation levels. The compensation levels can be determined by rule templates: audio resources, text display resources, and enhanced quality resources attached to mandatory resources enter the first level; main motion resources and partial image quality resources enter the second level; static background resources and enhanced quality resources attached to static backgrounds enter the third level. Within the same level, the system prioritizes debts with tighter expiration dates and smaller compensation data sizes, ensuring that limited budgets are used to restore more critical audio and video resources.

[0067] The selection of the debt compensation set employs set selection, hierarchical sorting, and capacity constraint rules. The initial source of set selection is subset selection from mathematical set theory, while the capacity constraint originates from the engineering rule in communication scheduling that "the amount of data to be sent does not exceed the available sending budget." This application combines these two principles into the experiential debt compensation scenario, starting with the experiential debt set... The system filters out still valid debt records, then forms an ordered compensation queue based on compensation level, validity period, and compensation data size. Subsequently, under the condition that the cumulative compensation data size does not exceed the compensation budget, each debt is selected into the set of debts that can actually be sent in the current period. ; in, This represents the set of experience debts selected for the currently compensable transmission cycle; Indicates the set of experiential debts The set of valid experience debt records selected from the data. Each item in the table satisfies the following conditions: the debt status is pending or partially compensated, the effective period has not yet expired, and the compensation data index is readable. express Sort by compensation level, validity period, and compensation data size. Experience debt records; Indicates that the sorting position is no later than And has already been selected Experience debt records; Indicates compensation for debt records The required data size is read from the data packet pointed to by the compensation data index, the enhancement layer data, or the unsent data range; it represents the budget allocated to compensation transmission in the current compensable transmission cycle, determined by the transmission budget for that cycle. This is obtained after deducting the normal mandatory transmission tasks. The derivation process of this formula is as follows: First from... Screening effective debts yields Then, according to the compensation level, validity period, and compensation data size, Sort, and finally add items one by one according to the sorting results. And ensure that the cumulative size of the compensation data for the selected debts does not exceed [the specified value]. In the formula and All are objects that can send data, and the cumulative constraint can ensure that multiple compensation transmission tasks are still included in the current period's compensation budget after being merged.

[0068] In practical implementation, It is not a pre-existing static set, but rather constructed item by item by the system according to the sorting results. The system first... Set it to an empty set, then read the sorted sets sequentially. If this is added The cumulative compensation data size still does not exceed Then Write And update the cumulative value; if it exceeds after being added. If the debt is not selected, the selection process skips the current debt and proceeds to the next item, or the current debt selection is terminated according to the rule template. This implementation means that the "selected" clause in the formula... "It has a clear execution order, avoiding being interpreted as a loop definition."

[0069] In the compensation process for online education segmentation, assuming that step S103 outputs... The audio and whiteboard text have been sent. The teacher's main motion was sent in medium quality. The high-definition edge-enhanced whiteboard text was delayed and not sent. The background texture enhancement is currently omitted. This step generates three experience debt records: the teacher's main motion corresponds to the quality difference debt, the high-definition edge-enhanced whiteboard text corresponds to the delayed sending debt, and the background texture enhancement corresponds to the currently omitted debt. After entering a subsequent compensable transmission cycle, the system reads the compensation budget. Assume the supplementary data size for the teacher's main motion is 60, the data size for the high-definition edge enhancement of the blackboard is 30, and the data size for the texture enhancement of the background wall is 80, and all three are within their validity period; if If the value is 70, the system first sorts the data according to the compensation level. High-definition edge enhancement debt for the whiteboard text, being dependent on the whiteboard text display resources, enters a higher compensation level and is sorted first. Debt related to the motion quality difference of the teacher's main subject is sorted later, while background texture enhancement debt is at a lower compensation level. The system first selects the high-definition edge enhancement debt for the whiteboard text and writes it into the compensation transmission task, accumulating a compensation data size of 30. Then, it judges the motion quality difference debt for the teacher's main subject; if more is added, the cumulative compensation data size will reach 90, exceeding the limit. Therefore, this debt remains pending compensation. The data size of the background texture enhancement debt is 80, which exceeds the current compensation budget even when added alone, and the compensation level is low. Therefore, it remains pending compensation or is set to be abandoned after the expiration of the effective period. This calculation process shows that, under the same compensation budget, the system can prioritize the compensation of enhancement content related to key text resources according to the compensation level and cumulative budget constraints, so that subsequent transmission actions are consistent with the resource units and sacrificiality markers formed in the previous steps.

[0070] In the compensation process for sports live broadcast segments, if the output of step S103 is... The system prioritizes sending the commentary audio and scoreboard text, downgrades the sending of the main sports data, delays the sending of the high-definition enhanced scoreboard data, and omits the currently displayed audience background. This step generates liabilities related to the quality difference of the main sports data, the delayed sending of the high-definition enhanced scoreboard data, and the currently omitted audience background. In subsequent compensable transmission cycles, the high-definition enhanced scoreboard data liability, due to its dependence on the essential scoreboard text display resources, enters a higher compensation level; the sports data quality difference liability enters the main sports compensation level; and the omitted audience background liability enters a lower compensation level. When the budget allows, the system first sends the high-definition enhanced scoreboard data, providing users with a clearer score display during buffered playback or replay; subsequently, it compensates for the sports data quality difference, improving the clarity of continuous motion areas; the audience background can be set to forgo compensation after its expiration. This process transforms the delay, downgrade, and omission results generated in step S103 into actual compensation actions in this step, rather than merely retaining them as log records.

[0071] After the system generates the compensation transmission task, it merges the compensation transmission task with the regular audio and video transmission task for the same period into the transmission queue. For those selected... For experience debt, the system reads the corresponding compensation data packet based on the compensation data index and writes it to the sending queue; after sending, the status of the corresponding experience debt is updated to compensated. If only part of the compensation data packet is sent, the debt status is updated to partially compensated, and the sent boundary and the remaining compensation data index are recorded. If the experience debt is not selected in the current period but is still within its validity period, it remains in the pending compensation state. If the experience debt exceeds its validity period, or the corresponding resource type belongs to a low compensation value resource in the rule template, it is updated to expired or abandoned compensation. Through this status update, the experience debt set... It can be maintained continuously across multiple transmission cycles until the corresponding debt is compensated, partially compensated, expired, or waived.

[0072] This step outputs a set of experiential debts. and compensation transmission results . The set of actual transmission tasks in the current period The process includes downgraded transmission, delayed transmission, current omission, and partial transmission result generation. Each item is bound to the resource unit number, fragment number, debt type, target compensation quality, compensation priority, validity period, compensation data index, and debt status. This indicates the compensation transmission result completed within the compensable transmission period, and includes at least the resource unit number, fragment number, compensation data index, compensation transmission status, and debt status update result corresponding to the sent compensation data packets. Used to continuously record unfulfilled experiential debts. This is used to indicate that the selected experience debt has been compensated and sent, thus completing the closed loop from the actual transmission result of the current cycle to the subsequent compensation transmission action.

[0073] One embodiment of this application provides an adaptive allocation and transmission system for audio and video resources. For example... Figure 5 As shown, the system includes: The resource unit generation module is used to parse the audio and video segments to be transmitted to obtain candidate resource segments. It merges candidate resource segments with the same resource type, continuous playback time, and screen area or dependency relationship that meet the merging conditions to generate a set of audio and video resource units. Each audio and video resource unit includes resource unit number, segment number, time range, resource type, screen area range, target transmission quality, and dependency relationship. The sacrificiality marker generation module is used to generate sacrificiality markers for each audio and video resource unit based on resource type, dependency relationship, target transmission quality, time range, and screen area range. Sacrificiality markers include mandatory markers, degradeable markers, delayed markers, or omissionable markers. The actual transmission task generation module is used to convert each audio and video resource unit into a priority transmission task, a degraded transmission task, a delayed transmission task, or a currently omitted task within the current transmission cycle, based on the sacrificiality flag and the current transmission cycle transmission budget, to obtain the set of actual transmission tasks for the current cycle. The quality version selection module is used to select the quality version that meets the budget constraint from the available quality versions as the actual transmission quality version for the audio and video resource unit corresponding to the degradable mark. The experience debt generation module is used to generate experience debt records bound to resource unit numbers based on the actual processing method, target transmission quality, and actual transmission quality of each audio and video resource unit in the current cycle's actual transmission task set. The compensation transmission module is used to select the experience debt records to be compensated from the experience debt records and perform compensation transmission in subsequent compensateable transmission cycles, according to the compensation level, validity period and cumulative budget constraints.

[0074] It is worth noting that the specific workflow of the adaptive allocation and transmission system for audio and video resources provided in this embodiment of the invention is the same as that of the adaptive allocation and transmission method for audio and video resources described in the above embodiment, and will not be repeated here.

[0075] To illustrate the execution of this invention in a real audio / video transmission cycle, this embodiment tests and records a segment of online education. The playback time of this segment ranges from 0.000 seconds to 2.000 seconds. From the segment parsing results, the system obtains five audio / video resource units: teacher's audio, whiteboard text, teacher's main image, high-definition edge enhancement of the whiteboard text, and background texture. Current cycle transmission budget. The budget is 500 for subsequent compensation cycles. The value is 70; the test log only records the resource unit type, target quality, available quality version data volume, dependency relationship and budget. The sacrificiality flag, actual task, experience debt and compensation result in the table are calculated by the system according to the aforementioned rules.

[0076]

[0077] Figure 6 This shows five resource units from the same online education segment. arrive , , and The transmission relationship is as follows. It can be seen that the system does not simply reduce the bitrate of the entire segment, but first writes the audio and whiteboard text as essential resources into the sending queue. Then, within the remaining budget, it selects a medium-quality version for the teacher's main image, while recording the high-definition edge enhancement of the whiteboard as a delay debt and the background texture as a currently omitted debt. In subsequent compensation cycles, the system adjusts the compensation level and... The capacity constraints prioritize compensating for high-definition edge enhancement data attached to whiteboard text, rather than compensating for low-value background textures. This embodiment demonstrates that the present invention can transform the degradation, delay, and omission results of the current cycle into traceable experience debt, and perform targeted compensation in subsequent cycles according to resource value and compensation budget, thereby forming a cross-cycle adaptive allocation and transmission closed loop of audio and video resources.

[0078] This invention also provides an adaptive allocation and transmission device for audio and video resources, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the steps described in the above embodiment of the adaptive allocation and transmission method for audio and video resources, for example... Figure 1 The steps S1 to S4 described above; or, when the processor executes the computer program, it implements the functions of each module in the above system embodiments.

[0079] For example, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the adaptive allocation and transmission device for audio and video resources.

[0080] The adaptive allocation and transmission device for audio and video resources can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the adaptive allocation and transmission device for audio and video resources may also include input / output devices, network access devices, buses, etc.

[0081] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the adaptive allocation and transmission device for audio and video resources, connecting all parts of the device via various interfaces and lines.

[0082] The memory can be used to store the computer program and / or modules. The processor, by running or executing the computer program and / or modules stored in the memory, and by calling the data stored in the memory, realizes various functions of the adaptive allocation and transmission device for audio and video resources. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created according to the operation of the controller, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD card), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0083] The module integrating the adaptive allocation and transmission of audio and video resources, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0084] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0085] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An adaptive allocation and transmission method for audio and video resources, characterized in that, include: The audio and video segments to be transmitted are parsed to obtain candidate resource segments. Candidate resource segments with the same resource type, continuous playback time, and screen area or dependency relationship that meet the merging conditions are merged to generate a set of audio and video resource units. Each audio and video resource unit includes resource unit number, segment number, time range, resource type, screen area range, target transmission quality, and dependency relationship. Based on the resource type, the dependency relationship, the target transmission quality, the time range, and the screen area range, a sacrificial tag is generated for each audio and video resource unit. The sacrificial tag includes a mandatory tag, a degradeable tag, a delayable tag, or an omissionable tag. Within the current transmission cycle, based on the sacrificiality flag and the current transmission cycle transmission budget, each audio / video resource unit is converted into a priority transmission task, a degraded transmission task, a delayed transmission task, or a currently omitted task, to obtain the actual transmission task set for the current cycle. For the audio and video resource units corresponding to the degradable marker, select the quality version that meets the budget constraint from the available quality versions as the actual transmission quality version; Based on the actual processing method, target transmission quality, and actual transmission quality of each audio and video resource unit in the current period's actual transmission task set, an experience debt record bound to the resource unit number is generated. In subsequent compensable transmission cycles, compensation is sent to select the experience debt records to be compensated from the experience debt records according to the compensation level, validity period, and cumulative budget constraints.

2. The adaptive allocation and transmission method for audio and video resources according to claim 1, characterized in that, The merging of candidate resource segments that are of the same resource type, have continuous playback time, and whose screen regions or dependencies meet the merging conditions specifically includes: Determine whether the resource types of two candidate resource segments are the same; if they are not the same, retain them as different audio and video resource units. If the resource types are the same, the time interval is calculated based on the start and end timestamps of the two candidate resource segments. When the time interval does not exceed the preset time connection threshold, the playback time is considered to be continuous. For resource types with a defined screen area, determine whether the screen areas of two candidate resource segments overlap, are adjacent, or belong to the same preset screen grid group; for audio resource types, determine whether two candidate resource segments are located on the same audio track; for enhanced quality resource types, determine whether two candidate resource segments are attached to the same base resource or adjacent base resources. When the resource types are the same, the playback time is continuous, and the screen area or dependency relationship meets the merging conditions, the two candidate resource segments are merged into the same audio and video resource unit.

3. The adaptive allocation and transmission method for audio and video resources according to claim 1, characterized in that, The step of generating a sacrificiality marker for each audio / video resource unit specifically includes: Based on the resource types, basic sacrificiality tags are generated, wherein audio resource types and text display resource types are mapped to mandatory tags, main motion resource types and local image quality resource types are mapped to degradeable tags, enhanced quality resource types are mapped to deferred tags, and static background resource types are mapped to omission tags. For audio and video resource units of the enhanced quality resource type that have a dependency relationship, the sacrificiality flag of the dependent basic audio and video resource unit is read and overwritten: when the sacrificiality flag of the dependent basic audio and video resource unit is a mandatory flag, the sacrificiality flag of the enhanced quality resource type audio and video resource unit is set to a deferred flag; when the sacrificiality flag of the dependent basic audio and video resource unit is an omissionable flag, the sacrificiality flag of the enhanced quality resource type audio and video resource unit is set to an omissionable flag.

4. The adaptive allocation and transmission method for audio and video resources according to claim 1, characterized in that, The step of selecting the quality version that meets the budget constraint from the available quality versions as the actual transmission quality version specifically includes: Read the set of available quality versions of the audio and video resource units corresponding to the degradable marker, and sort the set of available quality versions from low to high according to transmission quality; Select a quality version that meets the budget constraint from the set of available quality versions as the actual transmission quality version. The budget constraint is that the sum of the data size corresponding to the selected quality version and the attached protection transmission amount does not exceed the remaining transmission budget of the current period. The attached protection transmission amount is the minimum compensation entry data amount of the latency enhancement resource that is attached to the mandatory resource and has a compensation priority not lower than the current degradeable resource in the current fragment.

5. The adaptive allocation and transmission method for audio and video resources according to claim 1, characterized in that, The experience debt record includes debt number, resource unit number, fragment number, debt type, target compensation quality, compensation priority, validity period, compensation data index, and debt status; the debt type includes quality difference debt, delayed transmission debt, currently omitted debt, or partially transmitted debt; wherein: For audio and video resource units that are actually processed through downgraded transmission, quality difference debt is generated; For audio and video resource units that are actually processed by delayed transmission, a delayed transmission debt is generated; For audio and video resource units that are currently omitted in the actual processing method, generate the currently omitted debt; For audio and video resource units whose transmission status is partially transmitted, a partial transmission debt is generated.

6. The adaptive allocation and transmission method for audio and video resources according to claim 1, characterized in that, In subsequent compensable transmission cycles, according to the compensation level, validity period, and cumulative budget constraints, the process of selecting experience debt records to be compensated from the experience debt records and performing compensation transmission specifically includes: Filter the set of valid experience debt records from the experience debt records, which are in a debt status of pending or partial compensation, whose effective period has not yet expired, and whose compensation data index can be read. The set of valid experience debt records is sorted according to compensation level, validity period, and compensation data size; The system selects items one by one from the sorted results and adds them to the current period's compensable experience debt set until the cumulative compensation data size exceeds the compensation budget for the current compensation transmission period.

7. The adaptive allocation and transmission method for audio and video resources according to claim 6, characterized in that, The compensation level is determined based on the resource type: Audio resource types, text display resource types, and enhanced quality resource types that are dependent on essential resources correspond to the experience debt records in the first compensation level; The experience debt records corresponding to the main motion resource type and the local image quality resource type belong to the second compensation level; The experience debt records corresponding to static background resource types and enhanced quality resource types attached to static background resources belong to the third compensation level.

8. The adaptive allocation and transmission method for audio and video resources according to claim 1, characterized in that, The execution of compensation transmission specifically includes: Read the corresponding compensation data packet according to the compensation data index in the experience debt record and write it into the sending queue; After sending, the debt status of the experience debt record will be updated to compensated; When only a portion of the compensation data packet is sent, the debt status of the experience debt record is updated to partially compensated, and the sent boundary and the index of the remaining compensation data are recorded. When the experience debt record expires, the debt status of the experience debt record will be updated to waiver of compensation.

9. The adaptive allocation and transmission method for audio and video resources according to claim 1, characterized in that, The method further includes: When the data packet of the audio / video resource unit corresponding to the mandatory protection mark exceeds the remaining transmission budget of the current period, it is divided and sent according to the encapsulation packet boundary, encoding packet boundary, or text display data boundary. The basic packet that can be decoded, played, or displayed by the receiving end is written first, and the incomplete transmission status of the audio / video resource unit is recorded in the actual transmission task set of the current period.

10. An adaptive allocation and transmission system for audio and video resources, characterized in that, include: The resource unit generation module is used to parse the audio and video segments to be transmitted to obtain candidate resource segments. It merges candidate resource segments with the same resource type, continuous playback time, and screen area or dependency relationship that meet the merging conditions to generate a set of audio and video resource units. Each audio and video resource unit includes resource unit number, segment number, time range, resource type, screen area range, target transmission quality, and dependency relationship. The sacrificiality marker generation module is used to generate sacrificiality markers for each audio and video resource unit based on the resource type, the dependency relationship, the target transmission quality, the time range, and the screen area range. The sacrificiality markers include mandatory markers, degradeable markers, delayed markers, or omissionable markers. The actual transmission task generation module is used to convert each audio and video resource unit into a priority transmission task, a degraded transmission task, a delayed transmission task, or a currently omitted task within the current transmission cycle, based on the sacrificiality flag and the current transmission cycle transmission budget, to obtain the set of actual transmission tasks for the current cycle. The quality version selection module is used to select, from the available quality versions, the quality version that meets the budget constraint as the actual transmission quality version for the audio and video resource unit corresponding to the degradable mark. The experience debt generation module is used to generate an experience debt record bound to the resource unit number based on the actual processing method, target transmission quality and actual transmission quality of each audio and video resource unit in the current period's actual transmission task set. The compensation transmission module is used to select the experience debt record to be compensated from the experience debt records and perform compensation transmission in subsequent compensateable transmission cycles, according to the compensation level, validity period and cumulative budget constraints.

Images