Song queue and accompaniment cache co-scheduling method based on reinforcement learning

Abstract

The application discloses a song queue and accompaniment cache cooperative scheduling method based on reinforcement learning and belongs to the resource scheduling field of intelligent audio playing equipment. In order to solve the problems of inaccurate accompaniment preloading sequence, song cutting waiting, repeated downloading and cache redundancy caused by the dynamic change of a playing queue in a household karaoke sound box, the application jointly determines preloading, bandwidth allocation and cache elimination instructions through an event-driven semi-Markov decision process, hierarchical reinforcement learning, a queue-cache heterogeneous graph and a resource constraint action mask, so that the technical effect of improving accompaniment cache hit and resource utilization stability is realized.

Description

Technical Field

[0001] This invention relates to the field of resource scheduling for intelligent audio playback devices, and more particularly to a collaborative scheduling method for song queues and accompaniment buffers based on reinforcement learning. Background Technology

[0002] Home karaoke speakers need to retrieve song backing tracks from local cache or the network during playback, and continuously perform playback, preloading, and cache management based on the song list. As multiple people request songs, interject, switch songs, and cancel operations increase in home entertainment scenarios, the order of the playback queue and the set of candidate songs will change frequently in a short period of time, and the download tasks of backing tracks, cache usage, and playback deadlines will also change accordingly.

[0003] Existing home karaoke speakers typically employ fixed priority, first-in-first-out (FIFO), or static caching strategies to handle backing track preloading and cache eviction. These strategies struggle to simultaneously utilize playback progress, queue position, user actions, backing track download status, cache capacity, network throughput, and user's historical song selection behavior. When a song is interrupted or switched in the queue, a fixed strategy may continue downloading backing tracks that are about to be canceled or will not be played in the short term, resulting in insufficient download of backing tracks that are actually close to playback, leading to increased playback delays and song switching latency.

[0004] Furthermore, existing strategies typically handle preloading order, download bandwidth allocation, and cache eviction separately, lacking joint feedback on cache hits, duplicate downloads, cache redundancy, bandwidth peaks, and playback continuity. Because download completion status, cache protection objects, and network throughput changes cannot be promptly incorporated into a unified decision-making process, the system is prone to repeatedly downloading evicted accompaniment objects or mistakenly deleting accompaniment objects about to be played when cache space is limited, thereby reducing the utilization of cache and network resources.

[0005] Therefore, a collaborative scheduling method for song queues and accompaniment buffers is needed to address the shortcomings of the existing technologies. Summary of the Invention

[0006] One objective of this invention is to propose a collaborative scheduling method for song queues and accompaniment caches based on reinforcement learning. Addressing the problem that existing technologies struggle to jointly consider playback status, queue changes, cache capacity, network status, and user history, leading to inaccurate accompaniment preloading order, song switching wait times, duplicate downloads, cache redundancy, and decreased resource utilization, this invention proposes a technical solution that uses an event-driven semi-Markov decision process, hierarchical reinforcement learning, queue-cache heterogeneous graphs, and resource constraint action masks to jointly generate accompaniment preloading, bandwidth allocation, and cache eviction instructions. This invention achieves the technical effects of improving accompaniment cache hit stability, reducing playback wait times and duplicate downloads, reducing cache redundancy, and smoothing network bandwidth usage.

[0007] This invention provides a reinforcement learning-based collaborative scheduling method for song queues and accompaniment buffers, comprising:

[0008] S1. Responding to the playback queue event of the home karaoke speaker, collect queue, accompaniment buffer, network transmission and user historical song selection data, and generate event-driven state records;

[0009] S2. Encode the event-driven state record according to the decision moment of the semi-Markov decision process to obtain a scheduling state vector containing playback progress, queue position, user operation event, accompaniment download status, cache occupancy status, network throughput and user's historical song selection behavior.

[0010] S3. Construct a queue-cached heterogeneous graph based on the scheduling state vector, calculate the playback deadline slack, playback demand score, preloading urgency score and cache eviction protection score of candidate songs, and obtain the scheduling features and feature validity flags of candidate songs.

[0011] S4. Input the candidate song scheduling features and feature validity flags into the hierarchical reinforcement learning strategy. The upper-level strategy outputs the preload quantity, cache space target and download bandwidth budget. The lower-level strategy selects the ordered preload song sequence, download rate allocation, cache eviction object and pending review field after filtering by the resource constraint action mask to obtain the collaborative scheduling action.

[0012] S5. Convert the collaborative scheduling action into automatic judgment instructions for accompaniment preloading, bandwidth allocation, cache eviction, download pause, cache retention or alarm exit. Construct reward values ​​based on cache hit, start-up delay, repeated download, cache redundancy, bandwidth peak and song switching delay after execution. Form experience samples and update the hierarchical reinforcement learning strategy when the update conditions are met.

[0013] Optionally, S1 includes:

[0014] The system monitors events such as song selection, song insertion, song switching, cancellation, playback progress refresh, changes in accompaniment download status, and network throughput refresh. The changes in accompaniment download status include events such as download task creation, download progress update, download pause, download resumption, download abortion, accompaniment download completion, and accompaniment download failure.

[0015] Each event is written into an event-driven state record, which includes the event type, event occurrence time, currently playing song identifier, queue number, candidate song identifier, backing track file length, number of bytes downloaded, number of bytes occupied in the cache, number of available cache bytes, throughput per unit time, and user history song request statistics fields.

[0016] When multiple download status records are generated for the same candidate song within the preset deduplication time window, the download status record with the latest timestamp is retained and the duplicate record is marked as a duplicate download candidate.

[0017] Optionally, S2 includes:

[0018] The events of song selection, insertion, song skipping, cancellation, change of accompaniment download status, and network throughput refresh are used as decision triggering events of a semi-Markov decision process. The change of accompaniment download status includes download task establishment, download progress update, download pause, download resumption, download abortion, accompaniment download completion, and accompaniment download failure events. The duration between adjacent decision triggering events is written into the status duration field.

[0019] The scheduling state vector is constructed based on the remaining duration of the currently playing song, the position of the candidate song queue, the download ratio of the candidate song's accompaniment, the cache usage ratio, the throughput per unit time, and the user's historical song request frequency.

[0020] The download ratio of the candidate song accompaniment is obtained by dividing the number of downloaded bytes by the length of the accompaniment file, the cache usage ratio is obtained by dividing the number of cached bytes by the cache capacity, and the user's historical song request frequency is normalized to the range of 0 to 1 according to the number of song requests within a preset historical time window.

[0021] Optionally, S3 includes:

[0022] Construct a queue-caching heterogeneous graph using candidate song nodes, queue position nodes, cache object nodes, download task nodes, and user history preference nodes;

[0023] Establish a playback order edge between the candidate song node and the queue position node, establish a cache hit edge between the candidate song node and the cache object node, establish a download status edge between the candidate song node and the download task node, and establish a historical demand edge between the candidate song node and the user's historical preference node.

[0024] Each edge carries an edge type, timestamp, normalized weight, and valid status field, and removes the playback order edge and download status edge corresponding to the canceled song;

[0025] Furthermore, the candidate song scheduling feature is calculated as follows: the expected start time of the candidate song is obtained based on the remaining playback time and the playback time of the effective queue songs before the candidate song, and the remaining download time of the accompaniment is subtracted from the difference between the expected start time of the candidate song and the current time to obtain the playback end time relaxation.

[0026] The playback demand score is obtained by weighting the user's historical song request frequency, the last item of the current queue position, and the insertion flag. The insertion flag is set to one when the candidate song corresponds to the latest valid insertion event in the status snapshot and has not been canceled or invalidated; otherwise, it is set to zero.

[0027] The preloading urgency score is obtained by weighting the inverse normalized term of the playback deadline relaxation, the proportion of undownloaded backing tracks, and the playback demand score.

[0028] The cache elimination protection score is obtained by weighting the current playback flag, the candidate queue position (which is less than the protection bit threshold and whose downloaded ratio is not less than the protection ratio threshold) numbered from zero and does not include the currently playing song, the upcoming playback protection flag, the historical repeat frequency, and the downloaded ratio.

[0029] The protection bit threshold represents the number of protected candidate songs to be played. The protection ratio threshold and the protection bit threshold are obtained by looking up a table together from the current network throughput tier and the cache usage ratio tier.

[0030] Optionally, S4 includes:

[0031] The upper-layer strategy adopts a semi-Markov option strategy with state duration as a discount factor. It reads the scheduling state vector, the candidate song scheduling features, the feature validity flag, the remaining cache capacity ratio, the network throughput tier, and the pre-generated set of releaseable cache candidates constructed by the prequeue-cache heterogeneous graph. It outputs the preload quantity, cache space target, and download bandwidth budget.

[0032] The value of the preload quantity does not exceed the number of candidate songs in the queue that have not completed the accompaniment download. The cache space target represents the net new cache occupancy allowed after the releaseable cache candidate set is included in this round of action. The value of the cache space target does not exceed the sum of the current available cache bytes and the conservative release bytes corresponding to the releaseable cache candidate set. The value of the download bandwidth budget does not exceed the current throughput per unit time multiplied by the preset bandwidth occupancy upper limit coefficient.

[0033] Furthermore, the actions of the lower-level strategy consist of an ordered preloaded song sequence, a download rate allocation value for each download task, a cache eviction object set, and a field to be reviewed and processed.

[0034] Before selecting an action, a resource constraint action mask is generated. The resource constraint action mask sets the business execution action that meets any of the following conditions as non-executable: the number of newly downloaded bytes determined by subtracting the number of downloaded bytes from the length of the accompaniment file and trimming it to a non-negative value, and the net new cache occupancy obtained after deducting the number of cache bytes that can be included in the cache release from the cache eviction object set in this round, exceeds the cache space target; the sum of the download rate allocation values ​​of each download task exceeds the download bandwidth budget; the cache eviction object set contains the accompaniment object of the currently playing song; the cache eviction object set contains the accompaniment object whose queue position is less than the protection bit threshold and whose downloaded ratio is not less than the protection ratio threshold; preloading or eviction is performed on candidate songs or cache objects in the review status; the feature validity constraint bit is non-executable; or the number of cache occupancy bytes exceeds the cache capacity after the action is executed.

[0035] Among them, download pause, cache retention or alarm exit security actions corresponding to the fields to be reviewed are not the filtered business execution actions;

[0036] When the set of business execution actions filtered by the resource constraint action mask is empty, a conservative action is output. The conservative action sets the ordered preloaded song sequence to empty, sets the newly added download rate allocation value to zero, and sets the cache eviction object set to empty. This branch is recorded as a mask conservative flag rather than a downstream execution failure flag.

[0037] Furthermore, the lower-level strategy sorts the candidate action reward values ​​in the action set after filtering by the resource-constrained action mask and selects the action with the highest reward value.

[0038] The candidate action benefit value is obtained by weighting the preload urgency score, cache eviction protection score, estimated start-up delay duration, estimated number of repeated download bytes, estimated number of cache redundant bytes, and estimated peak bandwidth utilization ratio. The estimated start-up delay duration is calculated based on the remaining download bytes of the candidate song, the download rate allocation value, and the estimated playback start time. The estimated number of repeated download bytes is calculated based on the number of accompaniment object bytes that will still be in the effective queue in the cache eviction object set in the future. The estimated number of cache redundant bytes is calculated based on the number of accompaniment object bytes that are not in the effective queue and are not protected by historical repeat after the action is executed. The estimated peak bandwidth utilization ratio is calculated based on the ratio of the sum of the download rate allocation values ​​of each download task to the estimated available throughput value frozen by the network monitor at the start of action selection.

[0039] The preload urgency score is used as a positive benefit, while the cache eviction protection score, the expected start-up delay, the expected number of duplicate download bytes, the expected number of cache redundant bytes, and the expected peak bandwidth usage ratio are used as penalty factors.

[0040] When the ordered preloaded song sequence or the cache elimination object set is empty, the corresponding average preload urgency or average cache elimination protection score is zero. When the candidate action benefit values ​​of multiple actions are the same, the action with the shorter expected start-up delay time is selected first.

[0041] Optionally, S5 includes:

[0042] First, the coordinated scheduling actions are converted into download task creation or recovery commands, token bucket rate parameters, and cache deletion commands. Then, after each instruction is executed, the cache hit flag, start-up delay duration, number of repeated download bytes, number of cache redundant bytes, peak bandwidth occupancy ratio, and song switching delay are recorded according to the status snapshot number, song identifier, and execution window attribution record in the action record table.

[0043] Positive reward items are generated based on the cache hit flag, and negative reward items are generated based on the start-up delay duration, number of repeated download bytes, number of cache redundant bytes, peak bandwidth usage ratio, song switching delay, and downstream execution failure flag. The mask conservative flag is used as a source of negative samples for unexecutable actions rather than as a downstream execution failure flag.

[0044] Write the scheduling state vector, the cooperative scheduling action, the reward value, the scheduling state vector at the next decision moment, and the state duration into the experience sample;

[0045] When the number of experience samples reaches the local update batch threshold, the experience samples are used to update the upper-level policy and the lower-level policy. When the number of experience samples does not reach the local update batch threshold, the experience samples are cached and the previous policy version is maintained or a conservative policy version is enabled.

[0046] Furthermore, the accompaniment preloading instruction, bandwidth allocation instruction, cache eviction instruction, and download pause instruction, cache hold instruction, or alarm exit automatic judgment instruction corresponding to the field to be reviewed are written into the download scheduler, cache manager, and alarm recorder of the home karaoke speaker;

[0047] The download scheduler establishes or resumes download tasks according to the order of the ordered preloaded song sequence, and adjusts the token bucket rate of concurrent download tasks according to the download rate allocation value. The cache manager deletes unprotected accompaniment objects according to the cache eviction object set and retains the accompaniment objects of the currently playing song and candidate song accompaniment objects whose queue position is less than the protection bit threshold and whose downloaded ratio is not less than the protection ratio threshold.

[0048] When a candidate song or cached object is written to the pending review and processing field, the download scheduler executes a download pause instruction on the corresponding candidate song, the cache manager executes a cache hold instruction on the corresponding cached object, or the alarm recorder executes an alarm exit automatic determination instruction according to the alarm reason code. The download pause instruction, the cache hold instruction, and the alarm exit automatic determination instruction are output as security processing independent of the business execution actions filtered by the resource constraint action mask.

[0049] When the network throughput is less than the bandwidth recovery threshold determined by the historical throughput percentile within a consecutive preset short time window, the priority of the low throughput pause branch is higher than the new preload instruction but lower than the download pause instruction, cache retention instruction and alarm exit automatic judgment instruction corresponding to the pending review and processing field. The download scheduler will set the new preload task to empty or postpone its execution, set the new download rate allocation value to zero and write it to the low throughput pause flag, and will not record the branch as a downstream execution failure until the next decision moment when the resource constraint action mask is regenerated.

[0050] The beneficial effects of this invention are:

[0051] 1. This invention uses changes in song selection, insertion, song switching, cancellation, accompaniment download status, and network throughput as event triggering sources for a semi-Markov decision process, enabling the scheduling status to be updated in real time with changes in the playback queue and network status, thereby reducing the response lag of fixed-period or fixed-priority strategies to queue mutations.

[0052] 2. This invention uses a queue-caching heterogeneous graph to express the relationship between candidate songs, queue positions, cache objects, download tasks, and user historical preferences, and calculates playback deadline slack, playback demand score, preloading urgency score, and cache eviction protection score, so that the accompaniment preloading order and cache protection object can simultaneously reflect the playback deadline, user demand, and cache status.

[0053] 3. This invention determines the preload quantity, cache space target, and download bandwidth budget through an upper-layer strategy, and selects an ordered preload song sequence, download rate allocation, and cache eviction object from the action set after filtering by a resource constraint action mask through a lower-layer strategy. This ensures that the reinforcement learning strategy makes decisions only within the action range that satisfies the cache capacity, bandwidth budget, current playback protection, and upcoming playback protection rules, thereby reducing the risks of accidental deletion, duplicate downloads, and peak bandwidth usage. Attached Figure Description

[0054] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0055] Figure 1This is a flowchart of a reinforcement learning-based collaborative scheduling method for song queues and accompaniment buffers.

[0056] Figure 2 This is a flowchart of step S4 of the present invention. Detailed Implementation

[0057] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0058] refer to Figures 1-2 A reinforcement learning-based collaborative scheduling method for song queues and accompaniment buffers includes:

[0059] S1. Responding to the playback queue event of the home karaoke speaker, collect queue, accompaniment buffer, network transmission and user historical song selection data, and generate event-driven state records;

[0060] S2. Encode the event-driven state record according to the decision moment of the semi-Markov decision process to obtain a scheduling state vector containing playback progress, queue position, user operation event, accompaniment download status, cache occupancy status, network throughput and user's historical song selection behavior.

[0061] S3. Construct a queue-cached heterogeneous graph based on the scheduling state vector, calculate the playback deadline slack, playback demand score, preloading urgency score and cache eviction protection score of candidate songs, and obtain the scheduling features and feature validity flags of candidate songs.

[0062] S4. Input the candidate song scheduling features and feature validity flags into the hierarchical reinforcement learning strategy. The upper-level strategy outputs the preload quantity, cache space target and download bandwidth budget. The lower-level strategy selects the ordered preload song sequence, download rate allocation, cache eviction object and pending review field after filtering by the resource constraint action mask to obtain the collaborative scheduling action.

[0063] S5. Convert the collaborative scheduling action into automatic judgment instructions for accompaniment preloading, bandwidth allocation, cache eviction, download pause, cache retention or alarm exit. Construct reward values ​​based on cache hit, start-up delay, repeated download, cache redundancy, bandwidth peak and song switching delay after execution. Form experience samples and update the hierarchical reinforcement learning strategy when the update conditions are met.

[0064] In this specific embodiment, S1 includes:

[0065] The scheduling controller of the home karaoke speaker establishes an event acquisition channel between the player, song queue manager, download scheduler, cache manager, and network monitor. This event acquisition channel uses the playback session number, song identifier, and monotonically increasing timestamp as a joint key to receive events such as song selection, insertion, song skipping, cancellation, playback progress refresh, accompaniment download status change, and network throughput refresh. Accompaniment download status changes include download task creation, download progress update, download pause, download resumption, download abort, accompaniment download completion, and accompaniment download failure events. Each event is written to an event log buffer. Each record in the event log buffer carries the event type, event occurrence time, currently playing song identifier, candidate song identifier, queue number, remaining playback time, accompaniment file length, number of downloaded bytes, number of bytes occupied in the cache, number of available cache bytes, throughput per unit time, and user historical song selection statistics fields.

[0066] The scheduling controller uses the current playback progress reported by the player as the playback status source, the candidate song list maintained by the queue manager as the queue operation data source, the cache directory maintained by the cache manager as the accompaniment cache data source, and the download byte increment counted by the network monitor within the sampling period as the network transmission data source. In this specific embodiment, the sampling period is set to one second, and the throughput per unit time is determined by… The calculation yielded, where This indicates the number of newly downloaded bytes within the sampling period. Indicates the sampling period. The unit is bytes per second, and the calculation result is written to the throughput field of the event-driven state record;

[0067] When a song request event is received, the queue manager writes the new song identifier, the requesting user identifier, the request time, and the expected length of the accompaniment file into the queue operation data, and generates an undownloaded status for the song. When an insertion event is received, the scheduler writes the queue number of the inserted song to the next position after the currently playing song, and synchronously updates the queue number of the affected candidate songs. When a song skipping event is received, the player writes the skipped song identifier, the switching time, and the next song identifier into the playback status field. The queue manager removes the skipped song from the valid queue and writes it into the skipped status. The download scheduler marks its incomplete download task as pauseable, and the accompaniment bytes already written to the cache directory are retained as a verifiable cache object. When a cancellation event is received, the queue manager writes the status of the candidate corresponding to the cancelled song as invalid, and marks the incomplete download task corresponding to that candidate as abortable.

[0068] The accompaniment download status is maintained by the download scheduler according to the task identifier. The task status field includes task identifier, song identifier, accompaniment file length, number of bytes downloaded, number of bytes remaining, download start time, last byte update time, and failure reason code. When multiple download status records of the same candidate song enter the preset deduplication time window, the scheduling controller selects the latest record as the valid download status record according to the timestamp and writes the remaining records into the duplicate download candidate list. The preset deduplication time window is set to three sampling periods in this specific embodiment.

[0069] The event log buffer is merged according to the song identifier and the time of the event occurrence. The merging rules are as follows: queue events for the same song have higher priority than download progress events; cancellation events and skipped song status have higher priority than song request events and interstitial events; download completion events only update the cache object status and do not restore the valid queue status of canceled or skipped songs. For skipped songs, the event log buffer retains its cache object status and historical repeat field, so that step S3 can determine whether the cache object belongs to a cache redundancy object or a historical repeat protection object. After merging, the scheduling controller writes a record version number, source component identifier, and field integrity flag to each event-driven status record, so that subsequent steps can identify whether the record comes from the player, queue manager, download scheduler, cache manager, or network monitor.

[0070] User historical song request data is provided by the speaker's local user profile table. The user profile table uses user ID and song ID as keys to record the number of historical song requests, the most recent song request time, the number of repeats, and the number of cancellations. When generating event-driven state records, the scheduling controller reads the number of historical song requests and repeats corresponding to the current session user, and sets the historical song request statistics field corresponding to the missing user ID to a zero value without deleting the candidate song, so that subsequent state coding can distinguish between unknown preferences and determined low preferences.

[0071] When the network monitor does not return a throughput refresh event in the current sampling period, the scheduling controller uses the throughput per unit time from the previous sampling period and marks the throughput source as a hold value. When there are no throughput refresh events for three consecutive sampling periods, the throughput per unit time is set to the tenth percentile of the throughput observed in the current session and written to a low confidence state. This low confidence state will not delete candidate songs, but will limit the download bandwidth budget for new preloading actions in step S4.

[0072] The scheduling controller performs consistency cleanup on the event log buffer. If the same song has a download completion event after a cancellation event, the cancellation event is retained as the basis for queue validity, and the download completion event is used only as the basis for updating the cache object status. If the accompaniment file length is zero or missing, the candidate song is written to the pending review status and the preloading action is stopped. After the cleanup is completed, the scheduling controller outputs a set of event-driven status records sorted by timestamp and stores the set in the status snapshot area for step S2 to read.

[0073] In this specific embodiment, S2 includes:

[0074] The scheduling controller takes the event-driven state record set output in step S1 as input, and uses events such as song request, insertion, song skipping, cancellation, change of accompaniment download status, and network throughput refresh as decision triggering events of a semi-Markov decision process. The change of accompaniment download status includes events such as download task establishment, download progress update, download pause, download resumption, download abort, accompaniment download completion, and accompaniment download failure. When any accompaniment download status change event or other decision triggering event arrives, a scheduling snapshot is frozen once. The scheduling snapshot includes the time of event occurrence, the currently playing song, the candidate song set, the cache directory, the download task table, the user's historical song request statistics field, and the previous decision time.

[0075] The state duration between two adjacent decision-triggered events is determined by The calculation yielded, where Indicates the timestamp of the event that triggered the current decision. Indicates the timestamp of the event that triggered the previous decision. In seconds, the scheduling controller writes the duration between adjacent decision-triggered events into the status duration field; when and When timestamps are identical or out of order, the scheduling controller uses one sampling period as... The event will be marked as timestamp corrected, and subsequent policy updates will use the corrected timestamp. Calculate the discount factor;

[0076] For each candidate song The scheduling controller constructs a song state subvector, which includes the playback queue position. Remaining playback time Expected start time of candidate songs Accompaniment file length Number of bytes downloaded Cache hit flag Current playback indicator Download failure indicator Historical song request frequency and frequency of repeating ,in and In bytes and In seconds, This only indicates a cache hit. The candidate song identifier is set to 1 when it is equal to the currently playing song identifier, otherwise it is set to 0. The song state sub-vector is written into the state matrix according to the valid order of the candidate songs in the queue.

[0077] Expected start time of candidate songs Generated by the scheduling controller when freezing scheduling snapshots, the calculation rule is as follows: ,in Indicates the current decision-making moment. This indicates the remaining duration of the currently playing song. This indicates that the candidate songs are in the valid queue. The previous collection of songs that does not include the currently playing song. Indicates the song The estimated playback duration; First, read the duration of the accompaniment or original vocals from the song's metadata. Second, read the median duration of the player's most recent complete playback record. If both are missing, use the median duration of the valid songs played in the current session. ,like If the song is still unavailable, the corresponding candidate song will be added to the pending review status and the generation of new preloading actions will be prohibited; for interruption, cancellation, and song switching events, the valid queue order will be updated first. Calculate again For queues with the same sequence number, the events are sorted from earliest to latest based on their occurrence time.

[0078] The scheduling controller performs normalization on the download and cache fields, and the download ratio of candidate song accompaniment is adjusted accordingly. Obtain and trim to the 0 to 1 range, the cache usage ratio is from Obtain and crop to the 0 to 1 range, where Indicates the number of bytes occupied by the cache. Indicates the cache capacity in bytes; when When less than one byte, Set it to zero and write the song state subvector into the state to be reviewed to avoid the state encoding that cannot be reproduced due to zero denominator;

[0079] The state matrix is ​​generated by first filtering out invalid songs, then arranging candidate songs in ascending order of queue number, and further arranging them in ascending order of event occurrence time when they have the same queue number; each row of the state consists of a sequence of fields. The scheduling controller writes the field sequence, field unit, and field normalization method into a status code description table, which is then linked to the scheduling state vector. Using the same state snapshot number ensures that subsequent policies read the same field order and numerical meaning;

[0080] The frequency of user song requests in history is normalized based on the number of requests within a preset historical time window. In this specific embodiment, the preset historical time window is set to thirty days, and the normalized value is determined by... The calculation yielded, where This indicates the current user's preference for songs in the user profile table. Number of song requests Indicates the historical frequency normalization benchmark. The 95th percentile of the number of times the same user requested a song. In this specific implementation, it is set to avoid zero-sample denominators. The larger the value, the stronger the user's desire to select the song within the history window;

[0081] The normalized value of the frequency of historical repetitions is from The calculation yielded, where This indicates the current user's viewing history for songs within the preset time window. Number of times to repeat, This represents the normalized benchmark for repeat frequency; when the user profile table has no historical records, and All values ​​are set to zero and a new user flag is written. This flag serves as the basis for not triggering the history repeat protection when the protection lock is released subsequently.

[0082] The scheduling controller encodes playback progress, queue position, user operation events, backing track download status, buffer usage status, network throughput, and user's historical song request behavior into a scheduling status vector. The scheduling state vector is obtained by concatenating a global state subvector and a song state matrix. The global state subvector includes the current playback progress percentage and the cache usage percentage. Available cache bytes, throughput per unit time The fields for current download task count, most recent user operation type code, and status duration. Each row of the song state matrix corresponds to a candidate song;

[0083] When the number of candidate songs exceeds the policy input limit, the scheduling controller calculates the candidate retention key. ,in Indicates the queue number. This indicates that the playback deadline relaxation amount estimated in step S2 (within the frozen scheduling snapshot) is the same as that in step S3; Depend on get, , ,in This represents the estimated download bandwidth budget for the previous version of the policy that has already taken effect. This indicates the number of candidate songs in the frozen snapshot that have not been downloaded and are not currently awaiting review. In this specific implementation, it is set to one thousand bytes per second; when When the scheduler controller does not generate a candidate retention key and sets the valid preloaded candidate set to empty, it only retains cached eviction candidate objects and adds them to the extended object table; when Take when it is not available according to The estimated value after equal division , , and All read the same frozen scheduling snapshot, and Cut off before being included in the calculation And write the cropping mark; then press Ascending sum Preserve ascending order The first candidate song is selected, and the remaining candidate songs are only cached elimination candidates and written to the extended object table. When the number of candidate songs is lower than the policy input limit, the scheduling controller fills the state matrix with an invalid song mask. The invalid song mask filters the corresponding preloading action in the action set of step S4. Songs that are skipped, canceled, or missing are excluded. Songs from the rollback source are not included in the set of valid candidate songs, but their cached objects can still be included in the extended object table according to the historical repeat protection or cache redundancy rules.

[0084] The scheduling controller processes the scheduling state vector before outputting. Perform field validation. If the playback progress ratio, cache usage ratio, or download ratio exceeds the range of 0 to 1, trim the song according to the boundary and write a trimming flag. If a candidate song lacks a queue position or song identifier, move the candidate song to the extended object table and prohibit it from participating in preloading. The validated scheduling state vector Status duration field The status code description table and the extended object table are used together as inputs for step S3.

[0085] In this specific embodiment, S3 includes:

[0086] The graph maintenance module reads the scheduling state vector output in step S2. The song state matrix and extended object table are constructed, and a queue-cache heterogeneous graph is built in the memory graph structure. The queue-cached heterogeneous graph The nodes include candidate song nodes, queue position nodes, cache object nodes, download task nodes, and user history preference nodes. The node fields include node type, song identifier, queue number, cache object identifier, download task identifier, validity status, timestamp, and source status snapshot number.

[0087] The graph maintenance module establishes playback order edges between candidate song nodes and queue position nodes, cache hit edges between candidate song nodes and cache object nodes, download status edges between candidate song nodes and download task nodes, and historical demand edges between candidate song nodes and user history preference nodes. Each edge carries edge type, timestamp, normalized weight, and valid status fields. The weight of the playback order edge is determined by... Calculations show that This indicates the zero-starting position of the candidate song in the valid queue. The larger the value, the earlier it appears in the playback order;

[0088] For cache hit edges, download status edges, and historical demand edges, the graph maintenance module adopts a normalized weight generation rule corresponding to the edge type: the normalized weight of a cache hit edge is generated by... Received, among which Indicates the cache directory for candidate songs The cache hit flag is set to 1 for a hit and 0 for a miss. Indicates the accompaniment download ratio; the normalized weight of the download status edge is... Received, among which The download failure flag indicates a value of 1 for failure and 0 for success; the normalized weight of historical demand edges is determined by... Received, among which This represents the normalized value of historical song request frequency. This represents the normalized value of the frequency of historical repetitions. This represents a boundary clipping function that clips the input to the range of 0 to 1. All weights are written to the normalized weight field of the corresponding edge, with values ​​ranging from 0 to 1. When a candidate song is canceled or skipped by a song skipping, the normalized weights of the playback order edge and download status edge are reset to zero, and the corresponding valid edge is deleted. When a cached object is missing, the normalized weights of the cache hit edge are reset to zero. When a download fails, the valid status of the download status edge is written as "failure". Zeroing out historical preference records means that historical requirements are retained even when the user's historical preference record is of low confidence. Based on low confidence coefficient Attenuation, the In this specific implementation, the value is set to 0.5, and the attenuated weight and low confidence flag are written together into the edge history table;

[0089] The heterogeneous graph is updated using an incremental versioning rule, and the graph maintenance module establishes a graph version for each state snapshot number. When the queue position, cache hit status, or download task status changes, only the affected candidate song nodes and their adjacent edges are updated, and the edge status of the previous version is written to the edge history table. The edge history table fields include the previous weight, current weight, reason for change, and type of change event, so that subsequent reward updates can trace whether the scheduling action was triggered by insertion, song switching, cancellation, or changes in network throughput.

[0090] For songs that have been cancelled, the graph maintenance module deletes their playback order edges and download status edges, while retaining cache hit edges as a source for subsequent cache redundancy judgment. For songs that have been skipped, the graph maintenance module deletes their playback order edges, writes the incomplete download status edge as a pause candidate status, and retains cache hit edges as a source for cache redundancy or historical repeat protection judgment. For songs that have failed to download, the graph maintenance module retains the download status edge but writes the valid status field as failed, so that step S4 prevents further allocation of download rate in the action mask. For songs that have been fully downloaded and are still within the protection bit range, the valid status of the cache hit edge is written as protected, to prevent step S4 from treating them as elimination targets.

[0091] The graph maintenance module calculates candidate songs. The remaining download time for the accompaniment ,in Indicates the length of the backing track file. Indicates the number of bytes downloaded. This indicates the number of remaining download bytes after non-negative pruning. This indicates that step S2 freezes the scheduling snapshot and effectively estimates the throughput of the candidate songs already determined. This indicates a zero-throughput protection setting, which is set to 1,000 bytes per second in this specific embodiment. The unit is seconds; when At that time, the map maintenance module does not calculate the preloaded purpose. and The candidate preload set is set to empty, but cache redundancy and cache eviction protection scores can still be calculated; the playback deadline relaxation is... get, This indicates the expected start time for the candidate songs to play. Indicates the current decision-making moment. The smaller the value, the closer the candidate song is to the download deadline; this coarse-ranking relaxation is only used for candidate features and preloading urgency. Step S4 freezes the available throughput estimate at the start of action selection. And allocate values ​​based on actual download speed. Recalculate the expected start-up delay so that multitasking concurrency and token bucket rate limiting can be corrected in action rewards;

[0092] Playback demand score is determined by The calculation yielded, where Indicates the historical frequency of song requests. This indicates the last item in the queue. This indicates an insertion flag; when the candidate song identifier matches the song identifier of the latest valid insertion event in the status snapshot, and this insertion event has not been overwritten by subsequent cancellation, song skipping, or queue failure events, ,otherwise When the insertion event fails, the graph maintenance module synchronously sets the insertion flag field of the candidate song node to zero. , and Assuming the demand weights are equal to the sum of the three, in this specific implementation, they are set to 0.30, 0.50, and 0.20 respectively. Values ​​are cropped to the range of 0 to 1. The larger the value, the stronger the demand for the candidate song to be actually played;

[0093] Preload urgency score by The calculation yielded, where This represents the normalized term of the inverse of the relaxation term. The normalization reference for the relaxation amount is 30 seconds, as specified in this specific embodiment. Indicates the percentage of instrumental music downloaded. Indicates the score required for playback. The larger the value, the higher the priority for preloading; the map maintenance module will... Write the preload urgency field to the candidate song node;

[0094] Cache eviction protection score is determined by The calculation yielded, where Indicates the current playback status. This indicates that a protective sign will be played soon. This represents the protection bit threshold and is defined as the number of protected candidate songs to be played in the candidate queue, numbered starting from zero. The currently playing song is... Individual protection and not included , Indicates the protection ratio threshold. This represents the normalized value of the frequency of historical repetitions. Indicates the percentage of instrumental music downloaded. The larger the value, the less likely the corresponding cached object should be evicted. The protection bit threshold and protection ratio threshold are obtained by looking up a table together with the current network throughput tier and cache usage ratio tier.

[0095] Network throughput tiers are determined by the median throughput per unit time over the most recent sixty sampling periods, while cache utilization tiers are determined by the current cache utilization ratio. It is confirmed that the graph maintenance module uses throughput tiers and cache usage ratio tiers as joint lookup keys to read the protection bit threshold table; when the effective throughput samples are less than twenty or the network monitor writes a low confidence state, the throughput tier is processed as a low throughput tier and a low sample backoff flag is written; when the throughput tier decreases and the cache usage ratio increases, the protection bit threshold table outputs a more conservative protection bit, the purpose of which is to prioritize the protection of the song to be played under the combination of weak network and high cache usage, rather than expanding the range of new preloads;

[0096] The protection bit threshold table is stored in the local scheduling parameter area. The table fields include throughput tiers, cache usage ratio tiers, and protection bit thresholds. Protection ratio threshold The system includes version number, effective time, calibration source, and rollback flag. Throughput is categorized into three ranges in this specific implementation: low throughput, medium throughput, and high throughput, corresponding to less than 1 terabyte per second, 1 terabyte to 3 terabytes per second, and greater than 3 terabytes per second, respectively. Cache usage is categorized into three ranges: low usage, medium usage, and high usage, corresponding to less than 0.60, 0.60 to 0.80, and greater than 0.80, respectively. The complete threshold table is as follows: Low throughput, low usage... and Low throughput occupancy time and Low throughput and high occupancy and Medium throughput with low occupancy and During the throughput, the time occupied and High throughput and high occupancy and High throughput and low occupancy and High throughput occupancy time and High throughput and high occupancy and ;in The value range is from one to three, indicating that the protection number starts from zero. The first candidate song will be played, therefore the judgment method adopts... , The value range is 0.60 to 0.80; the current decision cycle reads the frozen version that has taken effect at the start of the decision; updates generated by user session statistics are only written to the next version; when a table entry is not hit or a low sample rollback occurs, the conservative record corresponding to low throughput and high usage in the most recent version is used.

[0097] The graph maintenance module also considers heterogeneous graphs. Perform a consistency check on candidate song features. The check includes whether the candidate song node has one and only one valid queue position node, whether the download task node matches the same song identifier, whether the cache hit status of the cache object node is consistent with the cache directory, and whether the historical demand edge references the user's historical preference node of the current session user. If a candidate song lacks a valid queue position node, its playback deadline relaxation is set to pending review. If the download task node and the cache object node point to different accompaniment file lengths, the accompaniment file length that passes the cache directory check is used and a field correction flag is written.

[0098] The candidate song scheduling feature table also records the feature validity flag, historical demand edge low confidence flag, and pending review record index for each candidate song. Missing or low-confidence historical preference fields are not written to pending review; instead, they are used only as historical demand edge attenuation flags or low confidence flags, causing the user's historical song request statistics field to be set to zero or attenuated by an attenuation coefficient while still retaining the candidate song. When the accompaniment file length is missing, the download ratio cannot be calculated, the song identifier is untraceable, the cache object identifier is untraceable, or there is no rollback source at the expected playback start time, the corresponding feature validity flag is written to pending review. The pending review record table fields include status snapshot number, song identifier, cache object identifier, trigger reason code, trigger field, review subject, review result, release condition, rollback path, alarm reason code, and handling output. The review subject is either the system integrity review module or... The manual review queue has the following review results: pass, pass after completion, keep paused, prohibit preloading, prohibit elimination, and alarm. When the system integrity review module completes the accompaniment file length, song identifier, cache object identifier, or expected playback start time and passes the verification, the pending review status is lifted and the file is returned to the candidate preloading set or cache elimination candidate set in the next decision cycle. When the review result is keep paused or prohibit preloading, step S4 can only generate a download pause handling field. When the review result is prohibit elimination, step S4 can only generate a cache keep handling field. When the review result is an alarm, the pending review record table is written with an alarm reason code and an alarm exit flag. Step S4 can only generate an alarm exit automatic judgment handling field and prohibit the object from entering the preloading, bandwidth allocation, and cache elimination service execution action set.

[0099] The graph maintenance module will play the deadline relaxation amount. Playback demand score Preload urgency score and cache eviction protection score The candidate song scheduling feature table is written, which uses the status snapshot number and song identifier as a joint key and records the calculation version, weight version, and input field verification status of each score; the graph maintenance module also generates a releaseable cached candidate set before proceeding to step S4. The set consists of cached objects in the cache directory that are not protected by the current playback, are not protected by the upcoming playback, are not in a pending review state, and whose verification can be passed before deletion. A release flag is written to each object. and conservatively released bytes ,in In this specific implementation, the value is set to 0.90; subsequently, the candidate song scheduling feature table and the queue-cache heterogeneous graph are... Protection bit threshold Protection ratio threshold, insertion flag field, feature validity flag, reason for review field, index of records to be reviewed, and releaseable cache candidate set. The protection bit number caliber is passed to step S4.

[0100] In this specific embodiment, S4 includes:

[0101] The hierarchical strategy execution module reads the candidate song scheduling feature table and queue-cached heterogeneous graph output from step S3. Protection bit threshold and a set of releaseable cache candidates And read the scheduling state vector output in step S2. and status duration field At the start of action selection, the hierarchical policy enforcement module reads and freezes the available throughput estimate from the network monitor. and its estimated version number, the It is only used for the action benefits, download rate allocation, and bandwidth mask calculation in step S4, and does not write back the candidate song scheduling features already generated in step S3; the hierarchical strategy execution module includes an upper-level option strategy and a lower-level action strategy, and the upper-level option strategy determines the preload quantity for this decision. Cache space target and download bandwidth budget The lower-level action strategy selects an ordered preloaded song sequence, download rate allocation value, and cache eviction object set from the action set after filtering by the resource constraint action mask;

[0102] The upper-level option strategy uses the state duration field. For a semi-Markov option strategy with a discount factor, the discount factor is determined by... The calculation yielded, where This represents the base discount factor for a sampling period. This indicates the sampling period in step S1. To measure the impact of longer-duration states on reward decay, the input to the upper-level option policy includes the scheduling state vector. The remaining percentage of cache capacity, network throughput tiers, statistics on the urgency of candidate song preloading, and statistics on cache eviction protection scores;

[0103] The number of preloaded items output by the upper-level option strategy satisfy ,in This indicates the number of candidate songs in the queue that have not completed accompaniment downloads and are not currently awaiting review; cache space target. This indicates the net increase in cache usage allowed after including the releaseable cache candidate set in this round of actions, satisfying... ,in Indicates the number of currently available cache bytes. This indicates the conservative number of bytes to release corresponding to the pre-generated set of releaseable cache candidates in step S3, and this value has been bound to the state snapshot number before the upper-layer policy output; download bandwidth budget. satisfy ,in This represents the throughput per unit of time. This indicates the preset bandwidth usage limit coefficient, which is set to 0.80 in this specific implementation. Upper-layer outputs exceeding the boundary are clipped to the corresponding boundary by projection rules.

[0104] Lower-level action strategy constructs candidate actions Candidate actions Including ordered preloaded song sequences Download rate allocation values ​​for each download task and cached object evicting collection ,in The number of songs in the collection does not exceed Each In bytes per second The objects in the cache are from the cache directory and do not contain invalid cache records. When generating candidate actions, they are first sorted by preload urgency score. Generate an ordered preloaded song sequence from largest to smallest, then use cache eviction protection scores. Candidates were selected and eliminated from the smallest to the largest age group;

[0105] The generation of candidate preloaded song sequences first excludes canceled songs, skipped songs, songs awaiting review, songs that failed to download and have not reached the retry interval, and songs that are already fully cached. Then, it ranks the songs according to their preload urgency score. Sort in descending order and write the sort number as the download task startup order field; the retry interval By failure reason code Number of consecutive failures Determined together with the retreat parameter table, the calculation formula is: ,in The time is measured in seconds and read from the backoff parameter table according to the failure reason code. It is an exponential backslip factor, and in this specific embodiment, its value ranges from 1.5 to 2.0. In this specific implementation, three hundred seconds are used. The number of consecutive failures is pruned to 0-5; the backoff parameter table fields include table version, failure reason code, base interval, exponential factor, maximum interval, maximum number of failures, effective time, and miss backoff flag. When the failure reason code is missed, the base interval corresponding to "temporarily unavailable network" is used for ten seconds, and the backoff flag is written; the timer starts from the moment of the last download failure, when... If the song is not included in the candidate preloaded song sequence, it will re-enter the candidate set after the retry interval is reached, retaining the original number of downloaded bytes. The generation of the candidate cache eviction object set first excludes the currently playing accompaniment object. Objects, protection lock objects, objects to be reviewed, and objects that meet the requirements The accompaniment object is then processed according to the cache eviction protection score. Arranged in ascending order; this generation rule ensures that the candidate set of lower-level action strategies already has executable object boundaries before entering the action mask;

[0106] Furthermore, when a failed download of a song reaches the retry interval, the scheduling controller first verifies that the song has not been canceled, has not been skipped, the length of the accompaniment file passes the verification, the song identifier is traceable, and if a cached object already exists, the cached object identifier is traceable. If the verification passes, the download status is updated from failed to retryable, and if some downloaded bytes have existed and the download task can be resumed, it is synchronously written to the pending recovery status. The temporary non-executable bit formed by the failed status is cleared, allowing the song to enter the candidate preloaded song sequence and participate in download rate allocation. If the verification fails, the failed status is maintained or the process is transferred to the pending review status, and the preload and bandwidth allocation actions continue to be filtered by the action mask.

[0107] Resource constraint action mask by This indicates that when the candidate business action meets the condition of being unexecutable. ,otherwise Unexecutable conditions include ordered preloaded song sequences. Additional download bytes required Deducting the number of cache bytes that can be released in this round The net increase in cache usage was then obtained Exceeding the cache space target The sum of the download rate allocation values ​​for each download task exceeds the download bandwidth budget. Cache eviction object collection Includes the instrumental track of the currently playing song. Cache eviction object collection Includes satisfying The accompanying object, the business execution action to be reviewed, the addition of preloading or cache eviction of the object, the feature validity constraint bit being set to non-executable, and the number of bytes occupied by the cache after the candidate action is executed. Exceeding cache capacity ;in Represents cached objects The number of bytes actually occupied in the cache directory. Representation Object The flag that can be included in the release space in this round is that the release space is only included when the cache eviction object of the candidate action passes the protection bit check and is not an object to be reviewed; the download pause or cache retention corresponding to the review processing field is a safe processing action and will not be filtered out because the execution action of the review business is not executable;

[0108] The calculation result of the action mask is stored in bitmap form. The bitmap fields include space constraint bits, bandwidth constraint bits, current playback protection bits, upcoming playback protection bits, cache capacity constraint bits, feature validity constraint bits, pending review business execution constraint bits, and pending review security handling bits. When any business execution constraint bit is zero, the corresponding business execution action cannot be executed. For the pending review object, the hierarchical policy execution module writes the corresponding song identifier, cache object identifier, trigger reason code, review record index, alarm reason code, and handling type into the pending review handling field. The handling type is limited to three types: download pause, cache hold, or alarm exit automatic judgment. The pending review security handling bit is taken as one when the handling field is complete and the object identifier is traceable, which is used to allow step S5 to output the download pause instruction, cache hold instruction, or alarm exit automatic judgment instruction. When the handling type is alarm exit automatic judgment, the hierarchical policy execution module also writes the alarm exit flag. The alarm reason code field is set as non-executable, and the preload bit, bandwidth allocation bit, and cache eviction bit of the corresponding business execution action are set as non-executable. The hierarchical policy execution module writes the bitmap, pending review and processing field, and filtered reason of each candidate action into the action mask table. The action mask table is used as a negative sample source of non-executable actions in step S5, instead of being directly discarded.

[0109] The triggering conditions for automatic alarm exit determination are determined by objective anomaly records, including untraceable song identifiers or cache object identifiers, abnormal accompaniment file length and failed review, consecutive cache verification failures reaching a threshold, download failures exceeding a threshold, the same object appearing in mutually exclusive states in multiple state snapshots, and alarm results given by the system integrity review module or manual review queue. When any triggering condition is met, the hierarchical policy execution module writes the corresponding alarm reason code, prohibiting the object from entering the preloading, bandwidth allocation, and cache eviction service action set, and only allowing the output of the automatic alarm exit determination command and corresponding alarm record.

[0110] For each candidate action, the hierarchical strategy execution module first calculates the estimated quantity before action execution, and then... Defined as a set of download tasks that already exist or will be created in the candidate actions. Defined as a download task or cache object Mapped song identifiers; estimated start-up delay comparison uses a unified set of song identifiers. When multiple download tasks exist for the same song, only the task with the latest timestamp and which is not paused is retained. If the song has multiple valid rate allocation values, their sum is taken as the download rate. Paused or failed tasks will not participate. The download rate allocation value is generated by a deterministic projection rule, first calculating the rate weight. Then calculate the original rate. ;in In this specific implementation, 16 kilobytes per second is taken, which is far from the walking distance. Takes 16 kilobytes per second; when empty , or At that time, all new additions Set to zero, and in Insufficient write bandwidth reason code and low throughput pause sign This prevents the branch from entering the minimum rate projection interval; when And when there are valid candidate preloaded songs, let And projected as If after projection ,according to After scaling proportionally, press again Round down to the nearest integer, remaining bandwidth is calculated as follows: Complete the items in descending order, ensuring that the total number of items does not exceed a certain limit. The normalized value for expected start-up delay is from Get, when Empty time command The expected normalized value for repeated downloads is... The expected cache redundancy normalization value is obtained from It is obtained that the expected peak bandwidth utilization rate is from Obtain; among them This represents the set of songs that remain in the valid queue after the action has been executed. This indicates a baseline for the risk of duplicate downloads, when no mapping exists. When evicting cached objects , This indicates the number of bytes of the accompaniment object that is not in a valid queue and is not protected by historical repeats after the action is executed;

[0111] For candidate actions that pass the resource-constrained action mask, the lower-level action strategy calculates the reward value of the candidate action. ,in Indicates an ordered preloaded song sequence The average preloading urgency of candidate songs, if If empty ; Represents the collection of cached objects to be evicted. The average cache eviction protection score of objects in the middle, if If empty Conservative actions are directly written to the fixed income field. And it does not participate in the ranking of executable action rewards; to The values ​​are derived from the revenue weighting table and sum to one. In this specific implementation, they are taken as 0.30, 0.15, 0.20, 0.15, 0.10, and 0.10, respectively.

[0112] The candidate action benefit value This is a positive benefit item; the larger the value, the more urgent the preloaded objects are. , , , and This is a penalty term; the larger the value, the higher the risk of the protected object being mistakenly eliminated, delayed start-up, duplicate downloads, cache redundancy, or peak bandwidth usage. Therefore, it is entered into the benefit value with a minus sign in the formula. The policy execution module calculates according to... Sort candidate actions from largest to smallest. When there are multiple actions... Choose when they are the same Smaller movements, when If the same conditions apply, choose the action that caches fewer objects for eviction;

[0113] During the policy training phase, the lower-level action policy also saves the candidate action benefit value before being filtered by the action mask and the executable action benefit value after filtering, which is used to distinguish between policy bias error and resource constraint error. During the online execution phase of the device, the hierarchical policy execution module only sends the filtered executable actions to the download scheduler and cache manager, and writes the policy mode field into the action record table. The policy mode field takes one of three states: training, online execution, or conservative execution.

[0114] The hierarchical strategy execution module also maintains a strategy version table, which includes the upper-level strategy version number, lower-level strategy version number, weight effective time, action mask version number, reward weight version number, rollback version number, rollback trigger threshold, rollback statistics window, conservative percentile value, recovery threshold, and version effective status. In this specific embodiment, the rollback statistics window is taken as twenty consecutive decision windows, the rollback trigger threshold is that the proportion of execution failure samples is not less than 0.30, and the conservative percentile value is taken as the 20th percentile value of the cache space target and download bandwidth budget in the most recent 30 effective windows. When step S5 feedback shows that the rollback trigger threshold is reached within the rollback statistics window, the hierarchical strategy execution module switches the online execution version of the next decision cycle to the rollback version and restricts the upper-level strategy output to the conservative percentile value. The recovery condition is that the cache hit rate is not less than 0.80 and the normalized mean of the start-up delay is not greater than 0.20 in the most recent 20 decision windows. The rollback or recovery result is only written to the version effective in the next decision cycle.

[0115] When all candidate actions are filtered by the resource-constrained action mask, the hierarchical strategy execution module outputs a conservative action, which will preload the song sequence in an orderly manner. Set to empty, set all newly added download rate allocation values ​​to zero, and reset the cache eviction object set. Set to empty; this conservative action is written to the mask conservative flag. Includes a mask for the reason list, but does not write it to the downstream execution failure flag. This allows step S5 to reduce the sampling probability of non-executable actions during policy updates, while also avoiding misrecording the safe and conservative output as a download failure or cache deletion rejection.

[0116] The hierarchical strategy execution module writes the final selected collaborative scheduling action into the action record table. The action record table fields include the status snapshot number and the upper-layer output. , and Ordered preloaded song sequence, preload priority number, download rate allocation value, cache eviction object set, fields to be reviewed and processed, alarm exit flag. Alarm reason code, insufficient bandwidth reason code Resource-constrained action mask results, candidate action reward values, Freeze value and estimated version number, policy mode field, mask conservative flag Low throughput pause indicator and downstream execution failure flags Then, the coordinated scheduling action and action record table are passed to step S5, and step S5 generates the accompaniment preloading instruction, bandwidth allocation instruction, cache eviction instruction, download pause instruction, cache retention instruction and alarm exit automatic judgment instruction.

[0117] In this specific embodiment, S5 includes:

[0118] The reward update module reads the collaborative scheduling actions and action record table output from step S4, first executes the downstream instruction mapping, and then preloads the song sequence in an orderly manner. Mapping to download task creation or resumption commands and their startup order, and assigning download rate values. Mapped to the token bucket rate parameter, this sets the cached objects to be evicted. Mapping is done to a cache deletion command, and the pending review processing field is mapped to a download pause command, cache retention command, or alarm exit automatic judgment command; when the processing type of the pending review processing field is alarm exit automatic judgment, the reward update module will... Set to 1, write the alarm reason code, status snapshot number, song identifier, and cache object identifier to the alarm logger, and stop the automatic preloading, bandwidth allocation, and cache eviction determination of the object in this execution window; if the action log table or network monitor writes a low throughput pause flag before execution. If the low-throughput pause branch has a higher priority than the newly added preload command but lower than the pending download pause command, pending cache retention command, and alarm exit auto-judgment command, the reward update module will rewrite the newly added preload command as an empty command or a delayed command, set the newly added download rate allocation value to zero, and retain the download pause command, cache retention command, alarm exit auto-judgment command, and the protected accompaniment object that has been downloaded. After the mapping is completed, the action log table will simultaneously save the original action field, the actual executed action field, the pending review field, the low-throughput pause flag, and the alarm exit flag. The alarm reason code and instruction field ensure that the reward update module can reverse-engineer the policy action based on the actual execution result. Subsequently, after the download scheduler, cache manager, alarm recorder, and player complete the execution of this instruction, the execution result is collected according to the status snapshot number, song identifier, and execution window number. The execution result includes cache hit flag, playback delay duration, number of repeated download bytes, number of cache redundant bytes, peak bandwidth occupancy ratio, song switching delay, and mask conservative flag. Low throughput pause indicator Alarm Exit Sign Downstream execution failure flag and the scheduling state vector at the next decision moment;

[0119] The cache hit flag is detected by the player when a candidate song enters the playback preparation stage. It is set to one if the corresponding accompaniment object is fully present in the local cache and the checksum passes; otherwise, it is set to zero. The playback delay duration is obtained from the time difference between the player requesting the accompaniment and the accompaniment becoming playable. The number of repeated download bytes is obtained by accumulating the byte increments of the same song identifier after it has been eliminated and then downloaded again. The number of cache redundant bytes is accumulated from the number of accompaniment objects that are not in the effective queue and are not protected by the user's history repeat function. The peak bandwidth utilization ratio is obtained by dividing the maximum download throughput within the execution window by the estimated available throughput from the network monitor. When no playback preparation or song-switching event has occurred in the current execution window, the corresponding delay field is set to an observation state, and the cache hit flag is set to zero. When no playback preparation occurs, a null value is also written, and the reward record table is synchronously written to the cache hit observation mask. , Startup observation mask Heqie Song Observation Mask The sample to be observed enters the backfill buffer but not the current strategy parameter update batch. It will be backfilled into the original action record table after a subsequent window generates an attributable playback preparation or song-switching event, based on the song identifier, status snapshot number, and execution window number. In this specific implementation, the backfill period is set to ten decision windows; if the sample is not observed within the specified period... The start-up delay or song-switching delay is written to a neutral value of zero and the timeout flag is retained. Timeout samples do not participate in cache hit rewards or gradient updates of start-up delay or song-switching delay, but only participate in conservative actions and mask negative sample statistics.

[0120] Reward value is determined by The calculation yielded, where Indicates a cache hit flag. This represents the normalized value obtained by dividing the measured start-up delay by a base delay in the same unit and truncating it to a value between 0 and 1. This represents the normalized value after dividing the actual number of repeatedly downloaded bytes by the baseline number of accompaniment bytes and truncating it to a value between 0 and 1. This represents the normalized value after dividing the measured number of redundant bytes in the cache by the cache capacity baseline and truncating it to a value between 0 and 1. This indicates the percentage of peak bandwidth utilization as measured. This represents the normalized value obtained by dividing the measured song switching delay by a reference value of the same unit and truncating it to the range of 0 to 1. This indicates a failure in execution. Items with positive rewards are listed; all other items have negative rewards.

[0121] The reward weights are derived from a reward weight table, whose fields include weight version, cache hit weight, start-up delay weight, duplicate download weight, cache redundancy weight, peak bandwidth weight, song switching delay weight, execution failure weight, and effective time. In this specific implementation, to The values ​​are 0.30, 0.18, 0.14, 0.12, 0.10, 0.12, and 0.04, respectively, and their sum is one. The current decision cycle reads the weighted version that has taken effect, and the weight update result is only written to the next version.

[0122] Normalized value of start delay duration is The calculated and truncated values ​​are in the range of 0 to 1, where This indicates the actual measured start-up delay time. This indicates the delay normalization baseline, which is set to five seconds in this specific embodiment. The zero protection term indicates the time in seconds. The normalized parameter table determines the time and is fixed at 0.001 seconds in this specific embodiment; the normalized value for the number of bytes downloaded repeatedly is determined by... The calculated and truncated values ​​are in the range of 0 to 1, where Indicates the number of bytes downloaded repeatedly. This represents the baseline for the risk of repeated downloads in actual measurements. This represents the median number of bytes actually cached from valid backing tracks in the cache directory; when there are no valid backing tracks in the cache directory... Take the median length of the backing track files involved in the current execution window. If the median is still unavailable, take the preset minimum backing track byte base. And write the low sample flag, In this specific implementation, it is set to one megabyte; This indicates a zero-protected entry in bytes. The parameter is determined by a normalized parameter table and is fixed at 4096 bytes in this specific embodiment; the normalized parameter table fields include parameter version, Values, The values, time units, byte units, effective time, and rollback flags are set, and rollback is set to 0.001 seconds and 4096 bytes respectively when an entry is not hit; the normalized value of the cached redundant bytes is determined by... The calculated and truncated values ​​are in the range of 0 to 1, where Indicates the number of redundant bytes in the cache. Indicates the cache capacity in bytes;

[0123] The peak bandwidth utilization rate and the normalized value of song switching latency are written into the reward field using the same truncation rule. The peak bandwidth utilization rate is determined by... The calculated and truncated values ​​are in the range of 0 to 1, where This represents the maximum download throughput observed within the action execution window. This represents the estimated available throughput that is frozen at the start of the decision-making process before the action is executed. Zero-protection entries represent throughput in bytes per second. and Uses the same bytes per second unit and the same sampling period length, but not the same time window; the reward record table is saved. Freeze time, estimated version number and The execution window number; the normalized value of the song switching delay is determined by... The calculated and truncated values ​​are in the range of 0 to 1, where This indicates the time difference between requesting to skip a song and the availability of the next instrumental track. The normalized reference for song switching delay is set to three seconds in this specific embodiment;

[0124] Available throughput The network monitor estimates the failure rate based on the number of valid downloaded bytes, the number of bytes retried due to transmission failures, and the number of tokens remaining in the token bucket during the ten sampling periods prior to the action execution. Then calculate the token bucket reserve coefficient. and calculate ,in This represents the median throughput per unit time over the ten most recent sampling periods before execution. This indicates the number of bytes retried for a failed request. Indicates the total number of bytes requested. This indicates the number of remaining tokens in the scheduler token bucket downloaded at the start of the decision-making process, expressed in bytes. This indicates a zero-protection entry in bytes; when the effective sampling period is less than three, the network monitor will... Set the estimated value to the 20th percentile of the historical throughput of this session and write a low sample flag. The estimated value remains frozen within the current action execution window.

[0125] The reward update module will schedule the state vector only when the reward field of the experience sample is closed and the observation mask allows for updates. Coordinated scheduling actions Reward Value The scheduling state vector at the next decision moment and state duration Write trainable experience samples and use a semi-Markov objective. Update the value function of the upper-level option policy, and update the action preferences of the lower-level action policy using action samples with resource-constrained action masks; when At that time, the mask reason is written as a negative sample feature into the non-executable action table but not deducted. ,when Write to the low-throughput pause field and set the positive update coefficient of the newly added preload action to zero. When writing the alarm exit sample field, alarm reason code, and manual review queue index, the corresponding object's automatic preloading, bandwidth allocation, and cache eviction actions are excluded from the positive action preference update as safe exit samples. Only then will the reward value be deducted from the downstream execution failure penalty item;

[0126] Experience samples are written to the experience replay buffer using the state snapshot number as the key. The sample fields include the scheduling state vector. Coordinated scheduling actions Actual action field, reward value The scheduling state vector at the next decision moment Status duration field Resource constraint action mask bitmap , , Alarm reason code Cache hit watch mask , Startup observation mask ,Song observation mask The backfill period, timeout flag, and execution result fields are included. The reward update module uses experience samples with closed reward and no timeout distortion to update the upper-level strategy and the lower-level strategy. The upper-level strategy updates the option value based on the reward error of the preload quantity, cache space target, and download bandwidth budget. The lower-level strategy updates the action preference based on the action benefit of the ordered preloaded song sequence, download rate allocation value, and cache elimination object set. Before backfilling, the observed samples only participate in the sample integrity statistics and do not enter the value loss and action preference gradient. Alarm exit samples only participate in the safety handling frequency and reason code statistics and are not considered as newly added preloaded successful samples.

[0127] When the upper-level policy is updated, the reward update module calculates the target value. The value loss is , and according to Update the upper-level value parameters, where The learning rate is set to a value and ranges from 0.0005 to 0.01 in the learning rate table. This represents a projection operator that clips parameters to the upper and lower bounds of the policy version table; when updating the lower-level policy, it uses the action preference function. Sum of policy probabilities This indicates the action selection rules, with the dominant quantity determined by... It is obtained that all resource-constrained action mask bitmaps are executable and and Action execution Positive updates; execution of filtered actions The negative update, characterized by spatial constraint bits, bandwidth constraint bits, current playback protection bits, upcoming playback protection bits, feature validity constraint bits, and pending review and processing constraint bits; The sample execution downstream failed negative updates. Failure characteristics come from download build failure, cache deletion rejection, or execution window anomalies. The learning rate table provides... , The range of values ​​for the failure update coefficient, and the projection interval is the action preference value. This reduces the probability of selecting unexecutable actions and downstream failure actions in subsequent decision-making.

[0128] The download scheduler creates or resumes download tasks based on the accompaniment preloading instruction and allocates values ​​according to the download rate. Set the token bucket rate for each task, and replenish the token bucket in each sampling period. Each byte token must be consumed before the task actually sends a request. When the network throughput is less than the bandwidth recovery threshold determined by the historical throughput percentile within a consecutive preset short time window, the download scheduler pauses the new preloading task. In this specific embodiment, the short time window is set to five sampling periods, and the bandwidth recovery threshold is determined by the 20th percentile of the throughput distribution in the most recent ten minutes. The execution priority of the low throughput pause branch is higher than the new preloading instruction, but lower than the current playback protection, the download pause instruction pending review, the cache hold instruction pending review, and the alarm exit automatic judgment instruction. When triggered, the new preloading sequence is set to empty or delayed, the new download rate percentile is configured to zero, and written to... Low throughput pause reason code, do not write And after regenerating the resource constraint action mask at the next decision moment, it can be either removed or continued;

[0129] The cache manager deletes unprotected backing tracks from the cache eviction object set according to the cache eviction command, and checks the currently playing song identifier and protection bit threshold before deletion. Protection ratio threshold And the download ratio of cached objects; if a cached object is evicted during execution, it meets the current playback flag. Or a protective sign will be played soon. If the object is not deleted, the cache manager will refuse to delete it and write the reason for the refusal to the action execution log. For accompaniment objects that have been downloaded and are within the protection range, the cache manager writes a protection lock, which is reviewed when the resource constraint action mask is regenerated at the next decision moment.

[0130] When the cache deletion command is rejected, the download task creation fails, or the execution window is abnormal, the reward update module marks this action as a downstream execution failure sample and writes the reason for the failure into the database. The corresponding execution failure fields; the mapping table fields from failure reason to temporary non-executable bit include failure reason code, bound song identifier, download task identifier, cache object identifier, action type, zeroed mask bit, scope of impact, freeze period, release trigger field, and re-evaluation formula; among them, download task establishment failure is bound to the song identifier and download task identifier, the zeroed mask bit is the preload action bit of the corresponding song, frozen for one decision period, and released when the download task status, network throughput tier, or retry interval status changes; cache deletion rejection is bound to the cache object identifier, the zeroed mask bit is the corresponding cache object eviction bit, frozen until the protection lock is released, pending review is passed, or the cache directory verification changes; execution window exception is bound to the action type and execution window number, the zeroed mask bit is the execution bit of the corresponding action type, frozen for one decision period, and released after the next window verification is passed; execution failure samples still enter the experience replay buffer, but the reward value is additionally deducted from the execution failure penalty item; conservative actions triggered by an empty resource constraint action mask are only written to Low throughput pauses branches for writing only Alarm exit automatic judgment branch only write The alarm reason code and none of the three factors trigger the downstream execution failure branch;

[0131] The protection lock release rule is executed by the cache manager at each decision moment. If the protected accompaniment object no longer belongs to the currently playing song, or if the protection flag is about to be displayed, the release rule will be executed. Furthermore, the normalized value of the frequency of historical repetitions Less than the protection release threshold If so, the protection lock is released and it is allowed to enter the cache eviction candidate set. In this specific implementation, it is set to 0.20; if the accompaniment object still meets any protection condition, the protection lock is maintained and the reason for maintaining it is written to the cache directory for the next round of action mask reading;

[0132] The reward update module performs sample priority sorting on the experience replay buffer, and the sample priority score is determined by... The calculation yielded, where Indicates the event of a song being switched. The resource constraint action mask takes one if a filter bit exists, otherwise it takes zero; if , or If the sample has not yet been backfilled, it will only remain in the backfill buffer and will not be included in the playback batch; samples are processed according to... The playback batches are entered from largest to smallest. When the scores are the same, samples with updated timestamps and closed reward fields are selected first. This allows the strategy to learn constraints related to song switching latency, playback continuity, non-executable resources, low throughput pause, alarm exit, and downstream execution failure.

[0133] When the number of experience samples is lower than the local update batch threshold, the scheduling controller only executes instructions and caches samples, without updating policy parameters. At the same time, it writes the policy update status as insufficient samples and maintains the previous policy version. In this specific implementation, the local update batch threshold is 64 experience samples, and the preset maximum hold window is 30 decision windows. If the insufficient sample status continues until the preset maximum hold window, the scheduling controller will activate the conservative policy version in the next decision cycle. The conservative policy version limits the number of newly preloaded items to one song, limits the download bandwidth budget to the 20th percentile of the most recent 30 valid windows, and prohibits the elimination of accompaniment objects within the protection bit threshold. The scheduling controller will exit the conservative policy version and write the recovery result to the next policy version only when the number of experience samples reaches the local update batch threshold and the most recent ten action execution logs do not contain download establishment failure or cache deletion rejection fields.

[0134] After the policy update is completed, the reward update module outputs the new hierarchical reinforcement learning policy version number, action record table version number, and cached scheduling log. The scheduling controller binds the policy version number with the next round of event-driven state records, enabling subsequent decisions to distinguish reward feedback under different policy versions. Cached samples participate in policy updates uniformly after reaching the batch threshold. The updated policy is first checked offline on the resource constraint action mask table, and only after the check passes is it written into the online execution version.

[0135] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0136] This invention utilizes an event-driven semi-Markov decision process to transform indefinite-interval queue changes, download state changes, and network throughput changes into learnable state durations and scheduling state vectors. This enables hierarchical reinforcement learning policies to update preload quantity, cache space target, download bandwidth budget, and lower-level actions at decision points that align with the playback event sequence. This approach ensures that playback waiting, repeated downloads, cache redundancy, and bandwidth peaks all enter the same reward feedback chain, avoiding local conflicts caused by separate optimizations of preloading, bandwidth allocation, and cache eviction.

[0137] This invention introduces a queue-cached heterogeneous graph at the policy input end and a resource constraint action mask at the policy output end. The former unifies playback order, cache hit, download status, and user historical preferences into candidate song scheduling features, while the latter excludes actions that exceed cache capacity, exceed bandwidth budget, violate current playback protection, or violate upcoming playback protection before action selection. This allows the reinforcement learning policy to explore and update within the set of executable actions, thereby achieving more stable coordinated scheduling between backing track preloading, download rate allocation, and cache eviction.

Claims

1. A song queue and accompaniment buffer collaborative scheduling method based on reinforcement learning, characterized in that, include: S1. Responding to the playback queue event of the home karaoke speaker, collect queue, accompaniment buffer, network transmission and user historical song selection data, and generate event-driven state records; S2. Encode the event-driven state record according to the decision moment of the semi-Markov decision process to obtain the scheduling state vector containing playback progress, queue position, user operation event, accompaniment download status, cache occupancy status, network throughput and user's historical song selection behavior. S3. Construct a queue-cached heterogeneous graph based on the scheduling state vector, calculate the playback deadline slack, playback demand score, preloading urgency score and cache eviction protection score of candidate songs, and obtain the scheduling features of candidate songs and the feature validity flag. S4. Input the candidate song scheduling features and feature validity flags into the hierarchical reinforcement learning strategy. The upper-level strategy outputs the preload quantity, cache space target and download bandwidth budget. The lower-level strategy selects the ordered preload song sequence, download rate allocation, cache eviction object and pending review field after filtering by the resource constraint action mask to obtain the collaborative scheduling action. S5. The coordinated scheduling actions are converted into automatic judgment instructions such as accompaniment preloading, bandwidth allocation, cache eviction, download pause, cache retention or alarm exit. Reward values ​​are constructed based on cache hit, start-up delay, repeated download, cache redundancy, bandwidth peak and song switching delay after execution. Experience samples are formed and the hierarchical reinforcement learning strategy is updated when the update conditions are met.

2. The song queue and accompaniment buffer collaborative scheduling method based on reinforcement learning according to claim 1, characterized in that, S1 includes: The system monitors events such as song request, insertion, song switching, cancellation, playback progress refresh, changes in accompaniment download status, and network throughput refresh. Changes in accompaniment download status include download task creation, download progress update, download pause, download resumption, download abortion, accompaniment download completion, and accompaniment download failure. Each event is written to an event-driven status record, which includes the event type, event occurrence time, currently playing song identifier, queue number, candidate song identifier, accompaniment file length, number of downloaded bytes, number of bytes occupied in the cache, number of available cache bytes, throughput per unit time, and user historical song request statistics. When multiple download status records for the same candidate song are generated within a preset deduplication time window, the download status record with the latest timestamp is retained, and duplicate records are marked as duplicate download candidates.

3. The song queue and accompaniment buffer collaborative scheduling method based on reinforcement learning according to claim 1, characterized in that, S2 include: The events of song request, insertion, song skipping, cancellation, change of accompaniment download status, and network throughput refresh are used as decision triggering events in a semi-Markov decision process. The change of accompaniment download status includes events such as download task establishment, download progress update, download pause, download resumption, download abort, accompaniment download completion, and accompaniment download failure. The duration between adjacent decision triggering events is written into the status duration field. The scheduling state vector is constructed according to the remaining duration of the currently playing song, the candidate song queue position, the candidate song accompaniment download ratio, the cache usage ratio, the throughput per unit time, and the user's historical song request frequency. The candidate song accompaniment download ratio is obtained by dividing the number of downloaded bytes by the length of the accompaniment file, the cache usage ratio is obtained by dividing the number of cached bytes by the cache capacity, and the user's historical song request frequency is normalized to the range of 0 to 1 according to the number of song requests within a preset historical time window.

4. The song queue and accompaniment buffer collaborative scheduling method based on reinforcement learning according to claim 1, characterized in that, S3 includes: constructing a queue-caching heterogeneous graph using candidate song nodes, queue position nodes, cache object nodes, download task nodes, and user history preference nodes; establishing playback order edges between candidate song nodes and queue position nodes, cache hit edges between candidate song nodes and cache object nodes, download status edges between candidate song nodes and download task nodes, and historical demand edges between candidate song nodes and user history preference nodes; each edge carries edge type, timestamp, normalized weight, and valid status fields, and deletes playback order edges and download status edges corresponding to canceled songs.

5. The song queue and accompaniment buffer collaborative scheduling method based on reinforcement learning according to claim 4, characterized in that, The candidate song scheduling features are calculated as follows: The estimated start time of the candidate song is obtained based on the remaining playback time and the playback time of the previous valid queue songs; the remaining download time of the accompaniment is subtracted from the difference between the estimated start time and the current time to obtain the playback deadline slack; a playback demand score is obtained by weighting the user's historical song request frequency, the inverse of the current queue position, and the insertion flag, wherein the insertion flag is set to one if the candidate song corresponds to the latest valid insertion event in the status snapshot and has not been canceled or invalidated, otherwise it is set to zero; the playback deadline slack is calculated as follows: The preloading urgency score is obtained by weighting the inverse normalized term of slack, the proportion of undownloaded accompaniment, and the playback demand score. The cache eviction protection score is obtained by weighting the current playback flag, the upcoming playback protection flag (which is numbered from zero and does not include the currently playing song, and whose position in the candidate queue is less than the protection bit threshold and whose downloaded proportion is not less than the protection proportion threshold), the historical repeat frequency, and the downloaded proportion. The protection bit threshold represents the number of protected upcoming playback candidate songs, and the protection proportion threshold and the protection bit threshold are obtained by looking up a table together from the current network throughput tier and the cache usage proportion tier.

6. The song queue and accompaniment buffer collaborative scheduling method based on reinforcement learning according to claim 5, characterized in that, S4 includes: The upper-layer strategy adopts a semi-Markov option strategy with state duration as a discount factor. It reads the scheduling state vector, the candidate song scheduling features, the feature validity flag, the remaining cache capacity ratio, the network throughput tier, and the pre-generated set of releaseable cache candidates constructed by the preorder queue-cache heterogeneous graph. It outputs the preload quantity, cache space target, and download bandwidth budget. The value of the preload quantity does not exceed the number of candidate songs in the queue that have not completed the accompaniment download. The cache space target represents the net increase in cache usage allowed after the releaseable cache candidate set is included in this round of action. The value of the cache space target does not exceed the sum of the current available cache bytes and the conservative release bytes corresponding to the releaseable cache candidate set. The value of the download bandwidth budget does not exceed the current throughput per unit time multiplied by the preset bandwidth usage upper limit coefficient.

7. The song queue and accompaniment buffer collaborative scheduling method based on reinforcement learning according to claim 6, characterized in that, The lower-level strategy's actions consist of an ordered preloaded song sequence, download rate allocation values ​​for each download task, a cache eviction object set, and fields for pending review and processing. Before selecting an action, a resource constraint action mask is generated. This mask sets any business action that meets any of the following conditions as non-executable: the net increase in cache usage, calculated by subtracting the number of downloaded bytes from the length of the accompaniment file and trimming it to a non-negative value, exceeds the cache space target after deducting the number of cache bytes that can be released from the cache eviction object set in this round; the sum of the download rate allocation values ​​for each download task exceeds the download bandwidth budget; the cache eviction object set contains the accompaniment object of the currently playing song; and the cache eviction object... The set contains accompaniment objects whose queue position is less than the protection bit threshold and whose downloaded proportion is not less than the protection proportion threshold. Candidate songs or cached objects in the pending review status are preloaded or evicted. The feature validity constraint bit is set to non-executable, or the cache occupancy exceeds the cache capacity after the action is executed. Download pause, cache retention, or alarm exit security actions corresponding to the pending review processing field are not filtered business execution actions. When the set of business execution actions filtered by the resource constraint action mask is empty, a conservative action is output. This conservative action sets the ordered preloaded song sequence to empty, sets the newly added download rate allocation value to zero, and sets the cache evicted object set to empty. This branch is recorded as a mask conservative flag rather than a downstream execution failure flag.

8. The song queue and accompaniment buffer collaborative scheduling method based on reinforcement learning according to claim 7, characterized in that, The lower-level strategy sorts the candidate actions in the action set after filtering by the resource-constrained action mask according to their reward values ​​and selects the action with the highest reward value. The reward value of the candidate action is obtained by weighting preload urgency score, cache eviction protection score, estimated start-up delay duration, estimated number of duplicate download bytes, estimated number of cache redundancy bytes, and estimated peak bandwidth usage ratio. Specifically, the estimated start-up delay duration is calculated based on the remaining download bytes of the candidate song, the download rate allocation value, and the estimated playback start time; the estimated number of duplicate download bytes is calculated based on the number of accompaniment object bytes still in the valid queue in the cache eviction object set; and the estimated number of cache redundancy bytes is calculated based on the number of bytes not in the valid queue after the action is executed. The number of bytes of the accompaniment object in the effective queue and not protected by historical repeats is calculated. The estimated peak bandwidth occupancy ratio is calculated as the ratio of the sum of the download rate allocation values ​​of each download task to the estimated available throughput value frozen by the network monitor at the start of action selection. The preload urgency score is used as a positive benefit, while the cache eviction protection score, estimated start-up delay, estimated number of repeated download bytes, estimated number of cache redundant bytes, and estimated peak bandwidth occupancy ratio are used as penalty factors. When the ordered preloaded song sequence or the cache eviction object set is empty, the corresponding average preload urgency or average cache eviction protection score is zero. When the benefit values ​​of multiple candidate actions are the same, the action with the shorter estimated start-up delay is selected first.

9. The song queue and accompaniment buffer collaborative scheduling method based on reinforcement learning according to claim 8, characterized in that, S5 includes: first, converting the collaborative scheduling action into download task creation or recovery commands, token bucket rate parameters, and cache deletion commands; then, after each instruction execution, according to the status snapshot number, song identifier, and execution window attribution record in the action record table, the cache hit flag, start-up delay duration, number of repeated download bytes, number of cache redundant bytes, peak bandwidth occupancy ratio, and song switching delay; generating positive reward items based on the cache hit flag, and generating negative reward items based on the start-up delay duration, number of repeated download bytes, number of cache redundant bytes, peak bandwidth occupancy ratio, song switching delay, and downstream execution failure flag, wherein the mask conservative flag serves as a source of negative samples for unexecutable actions and is not used as a downstream execution failure flag; writing the scheduling state vector, the collaborative scheduling action, the reward value, the scheduling state vector at the next decision moment, and the state duration into experience samples; when the number of experience samples reaches the local update batch threshold, using the experience samples to update the upper-layer strategy and the lower-layer strategy; when the number of experience samples does not reach the local update batch threshold, caching the experience samples and maintaining the previous strategy version or enabling the conservative strategy version.

10. The song queue and accompaniment buffer collaborative scheduling method based on reinforcement learning according to claim 9, characterized in that, The accompaniment preloading instruction, bandwidth allocation instruction, cache eviction instruction, and download pause instruction, cache hold instruction, or alarm exit automatic judgment instruction corresponding to the pending review and handling field are written into the download scheduler, cache manager, and alarm recorder of the home karaoke speaker; the download scheduler establishes or resumes download tasks according to the order of the ordered preloaded song sequence, and adjusts the token bucket rate of concurrent download tasks according to the download rate allocation value; the cache manager deletes unprotected accompaniment objects according to the cache eviction object set and retains the accompaniment object of the currently playing song and the accompaniment objects of candidate songs whose queue position is less than the protection bit threshold and whose downloaded ratio is not less than the protection ratio threshold; when a candidate song or cache object is written into the pending review and handling field, the download scheduler executes the download pause instruction for the corresponding candidate song, and the cache manager executes the download pause instruction for the corresponding candidate song. The corresponding cached object executes the cache retention instruction, or the alarm recorder executes the alarm exit automatic judgment instruction according to the alarm reason code. The download pause instruction, the cache retention instruction, and the alarm exit automatic judgment instruction are output as security actions independent of the service execution actions filtered by the resource constraint action mask. When the network throughput is less than the bandwidth recovery threshold determined by the historical throughput quantile value within a consecutive preset short time window, the priority of the low throughput pause branch is higher than the new preload instruction but lower than the download pause instruction, cache retention instruction, and alarm exit automatic judgment instruction corresponding to the pending review field. The download scheduler sets the new preload task to null or postpones its execution, sets the new download rate allocation value to zero and writes it to the low throughput pause flag, and does not record the branch as a downstream execution failure until the resource constraint action mask is regenerated at the next decision moment.

Images