Holographic page read scheduling method based on reinforcement learning

Abstract

This invention discloses a holographic page read scheduling method based on reinforcement learning, belonging to the field of optical storage medium data read scheduling and data center archive retrieval. To solve the problems of response latency, repeated addressing, and increased instantaneous power consumption in multi-angle holographic page reading, this invention constructs an angle-layer-page order heterogeneous addressing graph and combines graph neural networks, hierarchical reinforcement learning scheduling models, and security constraint action mask modules to generate read sequences and resource configurations, thereby achieving the technical effect of improving the adaptability of high-concurrency archive retrieval read scheduling.

Description

Technical Field

[0001] This invention relates to the fields of data reading scheduling for optical storage media and data center archiving and retrieval, and particularly to a holographic page reading scheduling method based on reinforcement learning. Background Technology

[0002] With the development of holographic storage media and data center archiving systems, multi-angle, multi-level, and multi-page sequence holographic page reading methods are gradually being used in scenarios involving the retrieval of large-capacity cold data and archived data. Existing holographic page reading systems typically determine the reading order based on the target page index, the current position of the read head, and fixed queue rules, and complete the data page reading through optical read heads, camera buffers, and laser power control.

[0003] However, in high-concurrency archive retrieval scenarios, the number of requests to be read is large and the target holographic pages are scattered. A fixed reading order is difficult to simultaneously take into account angle switching, layer movement, page order jump and device resource occupation, which can easily lead to repeated addressing and response waiting. At the same time, if the status of camera cache, laser power margin and historical error risk are not involved in the scheduling decision, it may lead to cache blocking, increased power consumption peak and increased error rereading, affecting the stability of batch reading.

[0004] Therefore, a holographic page reading scheduling method is needed to address the shortcomings of the existing technology. Summary of the Invention

[0005] One objective of this invention is to propose a holographic page read scheduling method based on reinforcement learning. Addressing the lack of adaptive scheduling for read order and resource consumption in existing multi-angle holographic page reads, which leads to high-concurrency archive retrieval response delays, duplicate addressing, and increased instantaneous power consumption, this invention proposes a technical solution that uses a heterogeneous addressing graph of angle-layer-page order for state encoding, and generates read windows, page read order, and resource configuration using a hierarchical reinforcement learning model. Reading is then executed after constraint filtering or action mask correction. This invention improves the adaptability of read scheduling and reduces invalid addressing and resource conflicts.

[0006] This invention provides a holographic page reading scheduling method based on reinforcement learning, comprising: S1, acquiring a concurrent archive retrieval request queue, a target holographic page index, and a reading device status including addressing location, media partition, camera buffer, laser power, bit error rate, and re-addressing records; S2, constructing a dynamic addressing cost graph based on the target holographic page index and the reading device status, wherein the dynamic addressing cost graph uses the holographic page to be read, angle, layer, page order, and device resources as nodes, and uses same-angle reading, layer switching, page order adjacency, resource conflict, and historical reread penalty as weighted edges; S3, inputting the dynamic addressing cost graph into a graph neural network to obtain an addressing cost representation and a queue congestion representation. S4. Input the addressing cost representation and queue congestion representation into the hierarchical reinforcement learning scheduling model to generate the access window, page reading order, batch size, exposure time, laser power level, and buffer release order. S5. Generate a safety constraint action mask based on the actuator movement range, camera frame rate limit, buffer remaining capacity, laser peak power, and read error rate threshold. Perform constraint correction on the page reading action to obtain the page reading sequence, resource occupancy plan, and power smoothing plan. S6. Drive the optical reader to read the holographic page according to the page reading sequence, resource occupancy plan, and power smoothing plan, and use the read feedback to update the reward function of the hierarchical reinforcement learning scheduling model.

[0007] Optionally, S1 includes:

[0008] Each archive retrieval request is parsed into a request number, target hologram page number, deadline timestamp, and read priority;

[0009] The target holographic page index is queried based on the target holographic page number to obtain the corresponding angle number, layer number, page sequence number and media partition number;

[0010] The ratio of camera cache occupancy to cache capacity is used as the cache occupancy rate, the ratio of laser power margin to laser rated power is used as the power margin rate, the historical read error rate and repeated addressing count are written into the read risk record according to a preset sampling period, and the request number, angle number, layer number, page sequence number, media partition number, cache occupancy rate, power margin rate and read risk record are combined to form a scheduling state vector.

[0011] Optionally, S2 includes:

[0012] Each holographic page to be read is represented as a page node. The page node includes an angle number, a layer number, a page sequence number, a media partition number, and a reading risk value. The reading risk value is obtained by weighting the historical reading error rate, the number of repeated addressings, and the media partition status according to a preset weight.

[0013] Page nodes with the same angle number are connected to the continuous reading edge of the same angle; page nodes with a layer number difference equal to the preset layer step size are connected to the adjacent layer switching edge; page nodes with a page sequence number difference equal to one are connected to the page sequence adjacent edge; page nodes that share at least one of the resources in the camera buffer and laser power channel are connected to the resource conflict edge; and page nodes with historical reread records are connected to the historical reread penalty edge.

[0014] The angle switching time, layer movement time, page order jump distance, cache utilization rate, power margin rate and the read risk value are normalized and weighted to obtain the addressing cost feature of the corresponding weighted edge, and the addressing cost feature is mapped monotonically to obtain the edge weight of the corresponding weighted edge.

[0015] Furthermore, the dynamic addressing cost graph is updated after each read operation is completed;

[0016] The angle switching time is obtained by querying the actuator calibration timetable from the current read head angle to the target angle;

[0017] The layer movement time is calculated from the displacement of the current layer to the target layer and the layer drive calibration speed;

[0018] The page jump distance is obtained by the difference between the current page number and the target page number;

[0019] The updated historical reread count, cache utilization, power margin, and read risk value are used to recalculate the addressing cost features of the corresponding weighted edges. The addressing cost features are then mapped in a monotonically decreasing manner to obtain the recalculated edge weights. The recalculated edge weights are then written into the dynamic addressing cost graph with a version number.

[0020] Optionally, S3 includes:

[0021] Embed node types, node state vectors, and edge weights into a message passing network for the input graph;

[0022] In each round of message passing, the adjacent node messages are multiplied by the corresponding edge weights according to the edge type and summed. The summation result is then concatenated with the node's own state and the node update vector is obtained through the parameter matrix and nonlinear function.

[0023] The page node update vector after a preset number of rounds is used as the addressing cost representation, and the queue congestion representation is obtained by concatenating the current queue length, the number of edges connecting the same resource, and the cache occupancy rate.

[0024] Optionally, S4 includes:

[0025] The hierarchical reinforcement learning scheduling model includes a window policy network and a page-level policy network;

[0026] The window policy network takes the addressing cost representation, the queue congestion representation, and the reading device status as inputs, and outputs the access window defined by the angle start and end number, the layer start and end number, and the page sequence start and end number;

[0027] The page-level policy network takes the page node representation and resource node status within the access window as input, and outputs the page reading order, the batch size, the exposure time, the laser power level, and the cache release order.

[0028] The batch size, the exposure time, and the laser power level are all selected from a preset discrete action table.

[0029] Optionally, S5 includes:

[0030] A set of possible actions is generated based on the actuator's movement range, the camera's frame rate limit, the remaining buffer capacity, the laser peak power, and the read error rate threshold.

[0031] Set the selection probability of page read actions that do not belong to the set of possible actions to zero;

[0032] When there is a page reading action that is set to zero in the page reading order, the page reading action that is set to zero is replaced by the action with the action that is ranked first in terms of action distance. The action distance is obtained by weighted summation of angle number difference, layer number difference, page order number difference, exposure time level difference and laser power level difference.

[0033] Generate the page read sequence, the resource allocation plan, and the power smoothing plan based on the replaced page read actions;

[0034] Furthermore, generating the power smoothing plan includes: calculating the power change based on the laser power level difference between two consecutive readout actions;

[0035] When the power change exceeds the preset power change threshold, without changing the page reading sequence, the laser power level of the next reading action is replaced with an adjacent power level that makes the power change not exceed the preset power change threshold, and the exposure time that meets the reading error rate threshold is selected according to the exposure level table.

[0036] Write the replaced laser power level and selected exposure time into the resource usage plan.

[0037] Optionally, S6 includes:

[0038] The read response time, the addressing movement distance, the number of repeated positioning, the peak power consumption, the cache blocking duration, and the read error feedback are normalized into response penalty items, movement penalty items, relocation penalty items, power consumption penalty items, blocking penalty items, and error penalty items, respectively.

[0039] The reward for a single action is obtained by subtracting the weighted sum of all penalties from the reward item corresponding to the number of pages successfully read.

[0040] The parameters of the window policy network and the page-level policy network are updated based on the single action reward.

[0041] Furthermore, during the offline training phase, training samples are constructed based on historical read logs. These training samples include a scheduling state vector, a dynamic addressing cost graph, an access window, page read actions, and single action rewards.

[0042] The cumulative reward for accessing the window is used as the training objective of the window policy network, and the single action reward for page reading is used as the training objective of the page-level policy network. The parameters of the window policy network and the page-level policy network are updated accordingly.

[0043] After training is completed, the updated parameters are used as the initial parameters for the online reading phase, and incremental updates are performed based on the single action reward obtained in step S6 during the online reading phase.

[0044] The beneficial effects of this invention are:

[0045] 1. By constructing a heterogeneous addressing graph containing angle, level, page order and device resource relationship based on the target holographic page index, and obtaining the addressing cost representation and queue congestion representation by graph neural network, the scheduling model can simultaneously perceive spatial addressing cost, request queue status and resource conflict risk, thereby reducing blind jumps and repeated positioning.

[0046] 2. By using a hierarchical reinforcement learning scheduling model, the angle-layer-page order reading window, as well as the page reading order, batch size, exposure time, laser power level, and cache release order within the window are output separately. This enables the reading scheduling to coordinate between global access range selection and local resource configuration, reducing response waiting and cache blocking under high-concurrency retrieval.

[0047] 3. By using a safety constraint action mask module or constraint filtering module to eliminate or correct actions that exceed the thresholds of actuator movement range, camera frame rate, buffer capacity, laser peak power, and bit error rate before action execution, and by combining the read feedback to update the reward function, the impact of instantaneous power consumption increase and bit error rereading on the reading process can be suppressed, thereby improving the stability of the batch reading process. Attached Figure Description

[0048] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0049] Figure 1 This is a flowchart of a holographic page reading scheduling method based on reinforcement learning.

[0050] Figure 2 This is a flowchart of the two-level decision-making process for window strategy and page-level strategy, and the empty window rollback process in step S4 of the present invention. Detailed Implementation

[0051] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0052] refer to Figures 1-2 A holographic page reading scheduling method based on reinforcement learning includes: S1, obtaining the concurrent archive retrieval request queue, the target holographic page index, and the reading device status including addressing location, media partition, camera buffer, laser power, bit error rate, and re-addressing records; S2, constructing a dynamic addressing cost graph based on the target holographic page index and the reading device status, wherein the dynamic addressing cost graph uses the holographic page to be read, angle, layer, page order, and device resources as nodes, and weighted edges for same-angle reading, layer switching, page order adjacency, resource conflict, and historical rereading penalty; S3, inputting the dynamic addressing cost graph into a graph neural network to obtain the addressing cost representation and queue congestion representation; S4 S1. Input the addressing cost representation and queue congestion representation into the hierarchical reinforcement learning scheduling model to generate the access window, page reading order, batch size, exposure time, laser power level, and buffer release order; S5. Generate a safety constraint action mask based on the actuator movement range, camera frame rate limit, buffer remaining capacity, laser peak power, and read error rate threshold to constrain and correct the page reading action, resulting in the page reading sequence, resource occupancy plan, and power smoothing plan; S6. Drive the optical reader to read the holographic page according to the page reading sequence, resource occupancy plan, and power smoothing plan, and use the read feedback to update the reward function of the hierarchical reinforcement learning scheduling model.

[0053] In this specific embodiment, S1 includes:

[0054] At the start of the scheduling cycle, the scheduling controller pulls the current concurrent archive retrieval request queue from the archive retrieval interface all at once, and parses the fields of each archive retrieval request in the queue to obtain the request number. Target holographic page number Deadline stamp and read priority ,in The sequence number of the request in the current queue is indicated. The deadline timestamp is used by the subsequent scheduling module to determine the timeout. The read priority is used by the subsequent scheduling module to sort the queue.

[0055] The scheduling controller maintains a target holographic page index table in local non-volatile memory, the target holographic page index table being organized by target holographic page number. As keys and numbered by angle Floor number Page numbering and media partition number The value is denoted by the angle number. The value ranges from 0 to 4095, and the floor number is [not specified]. The value range is from 0 to 255, and the page number is... The value ranges from 0 to 65535, representing the media partition number. The value ranges from 0 to 127. The scheduling controller obtains the target holographic page number through parsing. Then, the target holographic page index table is retrieved using a hash table and the corresponding index is returned. and And when the retrieval fails, the request is marked as missing index and removed from the current queue to avoid unexecutable actions in subsequent scheduling;

[0056] The device status is read synchronously from the camera control board, laser control board, and error statistics module and latched within the same scheduling cycle. The camera control board reports the number of bytes occupied by the camera buffer and the number of bytes of camera buffer capacity. The scheduling controller divides these two values ​​to obtain the buffer occupancy rate. Among them, cache utilization The value ranges from 0 to 1. The laser control board reports the laser power margin and the laser rated power. The dispatch controller divides the two to obtain the power margin rate. Among them, the power margin rate The value range is from 0 to 1;

[0057] The bit error statistics module uses a fixed sampling period. The historical read error rate and the number of repeated addressings are output and written into the read risk record. The historical read error rate is calculated by the number of erroneous bits and the total number of bits counted by the decoding and error correction module in each sampling period and recorded as a sequence. The number of repeated addressings is obtained by the number of repeated location events counted by the read head controller in each sampling period and recorded as a sequence. ,in To ensure that the historical length of risk records is read and overwrite is performed using a circular buffer, the reading of risk records always includes the most recent ones. Each sampling period and ;

[0058] After completing the index lookup and device status latch, the scheduling controller constructs a scheduling state vector for each request that has not been removed from the queue and writes it into the scheduling module input buffer. The scheduling state vector satisfies the following:

[0059] ;

[0060] in Indicates the first The scheduling state vector of each request. Indicates the request number. Indicates the angle number. Indicates the floor number. Indicates page number. Indicates the media partition number, Indicates cache utilization. Indicates the power margin rate. Indicates including the most recent A sequence of historical read bit error rates for each sampling period. Indicates including the most recent A sequence of repeated addressing times per sampling period, This indicates the length of the historical records read from the risk database.

[0061] In this specific embodiment, S2 includes:

[0062] The scheduling controller constructs a dynamic addressing cost graph based on the concurrent archive retrieval request queue and the status of the reading device. This dynamic addressing cost graph is denoted as... ,in The version number is the image number and is initially set to... For a set of nodes, Let be the set of edges. A set of edge types, For version number The set of edge weights;

[0063] The set of nodes Contains a set of page nodes Angle node set Layer node set Page sequence node set With device resource node set Each page node is associated with a target holographic page number. One-to-one correspondence and storage of node attributes ,in Number the angles. For floor numbering, Page numbering For media partition numbering, The risk value is to be read and its range is from 0 to 1;

[0064] The risk value of reading From step S1, the target holographic page number The bound read risk record is generated, and the read risk record includes the most recent... The scheduling controller averages the historical read bit error rate sequence and the repeated addressing count sequence for each sampling period to obtain the page-level average bit error rate, and then applies it to the upper limit. Normalization is performed by averaging the sequence of repeated addressing counts to obtain the page-level repeated addressing mean, and then adjusting the mean by the upper limit. Normalize and read the media partition status coefficients from the media partition monitoring module. and The value ranges from 0 to 1, and is then assigned a fixed weight. The risk value is obtained by weighting. The media partition state coefficient is used to characterize the readability and defect density of the media partition during the most recent inspection cycle, and the angle node set Each angle node in the hierarchy corresponds one-to-one with an angle number and is only used to aggregate page nodes with the same angle. With page order node set The device resource node set is used to aggregate page nodes with the same level and page order. At least include camera cache resource nodes With laser power resource nodes And it is used to express resource occupancy relationships;

[0065] The set of edges It contains five types of weighted edges and the set of edge types. Among them, the edges are read continuously at the same angle. Numbered by angle After grouping, sort the page nodes within the group by Sorting by lexicographical order and establishing bidirectional edges between adjacent page nodes, with layer switching edges. In satisfying Establish bidirectional edges and fix the layer step size. Page order adjacent edge In satisfying Establish bidirectional edges and resource conflict edges. This is used to express the shared use of camera buffer and laser power channel by page reads and through each page node and resource node. , A bidirectional edge is established between them to ensure that any two page nodes sharing the same resource node have resource conflict reachability in the graph structure, and historical rereads are penalized for edges. When a non-zero sample value exists in the sequence of repeated addressing times of a page node, a self-loop edge is created on that page node to explicitly inject historical reread penalties;

[0066] For page-to-page edges other than those involving resource conflicts, the scheduling controller calculates the addressing cost feature for each edge and maps it to an edge weight. The edge weight calculation satisfies the following:

[0067] ;

[0068] In the formula The version number is from page node Pointing to page node The right to the border, Represents an exponential function. Indicates the angle switching cost weight and takes Indicates the weight of the layer movement cost and takes Indicates the page order jump cost weight and takes Indicates the weight of cache usage cost and takes Represents the power margin cost weight and takes This indicates reading the risk cost weight and taking... The normalized value representing the angle switching time. This represents the normalized value indicating the time of stratum movement. The normalized value representing the page jump distance. This indicates the cache utilization rate of S1 latch. This indicates the power margin rate latched in step S1. and Representing page nodes respectively With page nodes Reading risk value, This indicates an operation that takes the larger value;

[0069] The angle switching time is determined by the timetable calibrated by the actuator. The result is obtained by directly querying the current readhead angle number and the target angle number, and then normalizing it by an upper limit of 15ms. The stratigraphic movement time is calculated by multiplying the difference between the current stratigraphic number and the target stratigraphic number by the stratigraphic drive calibration single-step time. Obtain and normalize according to the upper limit of 20ms. Page jump distance is Obtain and normalize to the upper limit of 1024. The normalization methods all employ truncation rules to limit the results to between 0 and 1;

[0070] After each read operation is completed, the scheduling controller performs an online update of the dynamic addressing cost graph and generates a new version number. The update process involves updating the version number to... The system reads the current head angle number, layer number, and page number from the head controller and writes them into the current device status. It also re-latches the buffer occupancy rate from the camera control board and laser control board. With power margin The error feedback corresponding to the reading action is read from the error statistics module and written into the reading risk record bound to the target holographic page number, thereby updating the reading risk value. Simultaneously, the re-addressing event count generated by the read action is accumulated into the re-addressing count sequence of the corresponding page node, and the historical reread penalty edge is updated accordingly. Existence;

[0071] During edge weight calculation, the scheduling controller recalculates the relationship between each page node in the current queue that has not yet been read and its adjacent page nodes. The corresponding edge in and And write the recalculated edge weights into the version number. edge weight set This allows the edge weights of the dynamic addressing cost graph to be updated in real time as the angle switching time, layer movement time, page order jump distance, cache utilization, power margin rate, and read risk value change.

[0072] In this specific embodiment, S3 includes:

[0073] The scheduling controller will have version number 1 Dynamic addressing cost graph Input a graph messaging network to generate addressing cost representations and queue congestion representations;

[0074] The graph message passing network employs a heterogeneous message passing structure with fixed parameters and performs a forward computation once in each scheduling cycle. The hidden dimension of the network is taken as... message passing rounds Nonlinear function take ;

[0075] For each node in the dynamic addressing cost graph Construct node type embeddings and node state vectors. The node type embedding is generated by a trainable lookup table matrix and used to distinguish page nodes, angle nodes, layer nodes, page sequence nodes, and device resource nodes. The node state vector is obtained by normalizing and concatenating the node's own attributes and storing it as a single-precision floating-point number. The page node state vector contains... And among them Number the angles. For floor numbering, Page numbering For media partition numbering, To read the risk value, For cache utilization, For power margin rate, the state vectors of angle nodes, layer nodes, and page sequence nodes include their corresponding numbers and global values. The state vector of a device resource node includes the occupancy or reserve of that resource and the global state. ;

[0076] The initial node representation is obtained by concatenating the node type embedding with the node state vector and then performing linear projection. And write the edge weights in step S2. It participates in message aggregation as an edge feature;

[0077] In the In round-robin message passing, the messages of adjacent nodes are weighted and summed according to edge type, and the node representation is updated. The update rule is as follows:

[0078] ;

[0079] in For message passing rounds and For the target node to be updated, For nodes In the The node representation after the round, This represents the vector concatenation operation. A set of edge types, For the type of The edge points to the node The set of adjacent nodes, For adjacent nodes, For version number slave node Pointing to node The right to the border, For edge type The corresponding trainable linear transformation matrix, To map the concatenated vector back A trainable linear transformation matrix in 3D space. It is an element-wise nonlinear function;

[0080] Finish After round-robin message passing, all page nodes Update vector The addressing cost is represented and composed of tensors in page node number order, which are then input into step S4.

[0081] Queue congestion is represented by the concatenation of the current queue length, the number of edges connecting to the same resource, and the cache utilization rate. The current queue length is obtained directly from the number of incomplete requests in step S1 and then increased to the upper limit. Normalized and truncated to 0 to 1, the number of edges connecting the same resource is obtained by counting the total number of resource conflict edges between all device resource nodes and page nodes in the dynamic addressing cost graph and then setting it to the upper limit. Normalized and truncated to 0 to 1, cache utilization is taken from S1 latch. The three are concatenated in a fixed order to form a queue congestion representation vector, which is then output together with the addressing cost representation to step S4.

[0082] In this specific embodiment, S4 includes:

[0083] The scheduling controller feeds the addressing cost representation and the queue congestion representation into the hierarchical reinforcement learning scheduling model to generate access windows and the page read order, batch size, exposure time, laser power level and buffer release order within the window;

[0084] The hierarchical reinforcement learning scheduling model consists of a window policy network and a page-level policy network, and the parameters of both remain the same set of updatable parameters during the online reading phase. The parameters of the window policy network are denoted as follows: Page-level policy network parameters are denoted as ,in and All are initialized during the offline training phase and incrementally updated according to the reward in S6;

[0085] The input to the window policy network consists of three concatenated parts. The first part is the addressing cost representation of all page nodes in the current queue, specifically the update vector for all page nodes. Perform mean pooling to obtain a global graph representation while preserving the dimensions. ,in In step S3, the first The first round of message passing Update the vector for each page node and The second part is the queue congestion representation vector obtained in step S3, and the third part is the reading device status vector, which includes the current read head angle number, the current layer number, the current page sequence number, and the cache utilization rate. With power margin ;

[0086] The window policy network is implemented using a two-layer fully connected network, with the first layer having an input dimension of... The output dimension is 128. The second layer output is a concatenated output of six action heads, corresponding to the angle start index, angle span index, layer start index, layer span index, page sequence start index, and page sequence span index, respectively. Each of the six action heads uses a linear layer to generate a discrete logarithmic probability vector of the action and then converts it into an action probability using softmax. The action probability calculation satisfies the following:

[0087] ;

[0088] In the formula Represents discrete actions The probability of choosing, This indicates the window policy network's response to actions. The output log odds, Represents an exponential function. This represents the discrete set of actions corresponding to this action head. Represents any candidate action in the discrete action set;

[0089] The discrete action set of the angle starting index is the index set of the deduplicated ascending sequence list of angle numbers appearing in the current queue, and the angle span index corresponds to a fixed angle span set. The discrete action set of the layer starting index is the index set of the deduplicated ascending sequence list of layer numbers appearing in the current queue, and the layer span index corresponds to the fixed layer span set. The discrete action set of the page order starting index is the index set of the deduplicated ascending sequence list of page order numbers appearing in the current queue, and the page order span index corresponds to the fixed page order span set. ;

[0090] The online reading phase window strategy network uses the maximum probability action as the output for each action head, maps the selected starting index to the starting number and calculates the ending number according to the selected span, thereby obtaining an access window jointly defined by the angle start and end number, the layer start and end number and the page sequence start and end number. The access window is used to filter page nodes in the current queue page node set that satisfy the angle number falling into the angle start and end number interval, the layer number falling into the layer start and end number interval and the page sequence number falling into the page sequence start and end number interval, and form the page node set within the window;

[0091] The page-level policy network takes the page node representation corresponding to the set of page nodes within a window and the resource node state as input, where the resource node state is obtained from the steps... The output camera cache resource node update vector and laser power resource node update vector are then compared with the device state vector. and splicing;

[0092] The page-level policy network is implemented using a shared coding and multi-head output structure. The shared coding is a two-layer fully connected network with an input dimension of [missing information]. The output has a dimension of 128, where the input to each page node within a window is the updated vector of that page node. The update vector of the camera cache resource node, the update vector of the laser power resource node, and the reading device status vector are concatenated together. The page reading order is obtained by outputting a page-level score for each page node in the window and sorting them from high to low scores.

[0093] Batch size is determined by the discrete action table. Output, exposure time via discrete action table Output, laser power level via discrete action table The output value is a normalized level of the laser's rated power.

[0094] The buffer release order is obtained by outputting the release priority score of each currently occupied camera buffer block and sorting them from high to low. The camera buffer blocks are fixed-size frame buffers, and the number of buffer blocks is fixed. Each cache block corresponds one-to-one with the page read result currently in the cache and is released according to the cache release order after decoding and verification to update the remaining cache capacity and make room for the next batch of page reads. The page read order, batch size, exposure time, laser power level and cache release order together constitute the action input of step S5.

[0095] In this specific embodiment, S5 includes:

[0096] The scheduling controller performs safety constraint masking and constraint correction on page read order and resource configuration, and generates page read sequence, resource occupancy plan and power smoothing plan after correction;

[0097] The scheduling controller first constructs a set of feasible actions based on the hard constraints of the reading device. The hard constraints include actuator movement range constraints, camera frame rate upper limit constraints, buffer remaining capacity constraints, laser peak power constraints, and read error rate threshold constraints, wherein the actuator movement range constraints are in single-step maximum angle steps. Maximum single-step layer step Maximum page sequence step in a single step This indicates and requires that the difference in angle number, layer number, and page number between two adjacent read actions does not exceed the aforementioned upper limit, and the camera frame rate upper limit constraint is as follows: Indicates and requests the exposure time for a single read action. Fixed overhead for camera readout The sum is not greater than The remaining cache capacity constraint is based on the number of currently free cache blocks. Indicate and require batch size Not greater than The camera cache is composed of It consists of several equal-sized cache blocks, with each cache block having a fixed size. Laser peak power constraint is based on normalized peak upper limit. Indicate and request the laser power level. Not greater than Read the bit error rate threshold constraint to It indicates and requires that the predicted bit error rate is no greater than ;

[0098] To achieve deterministic determination of the predicted bit error rate, the scheduling controller stores the bit error prediction calibration table in non-volatile memory. ,in For media partition numbering, The laser power level is taken from the discrete action table in step S4. , The exposure time is taken from the discrete action table of step S4. LUT The values ​​of the entries are determined during the equipment calibration phase for different media zones. and The error rate was obtained by combining the reading experiments and statistically analyzing the decoding results.

[0099] The scheduling controller represents each window inner page read action output by S4 as follows: ,in This represents the position number of the action in the page read order. Number the target page node and assign it the corresponding page node attributes For batch size, For the exposure time, This refers to the laser power level, followed by each... The above five types of hard constraints are used to determine the appropriate actions, and actions that satisfy all hard constraints are added. ;

[0100] During the action masking phase, the scheduler controller reads the selection probabilities of candidate actions from the page-level policy network and identifies those that do not belong to the target action. The action sets its selection probability to zero, then normalizes the remaining non-zero probability actions to keep the probability sum to 1, and performs replacement correction on actions whose probability is set to zero in the page read order. The replacement rule is as follows: Select the action with the smallest distance from the action to be replaced that has not yet been scheduled as the replacement action. The action distance is determined by the following formula:

[0101] ;

[0102] In the formula Indicates action With action Action distance, Indicates alternative actions, and Representing page nodes respectively With page nodes Angle numbering, and Representing page nodes respectively With page nodes The floor number, and Representing page nodes respectively With page nodes Page numbering, and Each represents an action With action Exposure time, and Each represents an action With action The laser power level, Indicates the exposure time level index and by Mapped from small to large Indicates the laser power level index and is pressed by Mapped from small to large , To fix the weights and to unify the differences between various dimensions, and to give higher priority to stratification movement and power fluctuation in the action distance;

[0103] After the replacement and correction are completed, the scheduling controller outputs the page read sequence according to the corrected action order and synchronously generates a resource allocation plan and a power smoothing plan. The resource allocation plan includes the batch size of each action. Mapped to the number of camera buffer blocks occupied and from the start of the action. deduction Each cache block is released according to the cache release order given in step S4 after the action is completed and the corresponding page data has been decoded and verified. At the same time, the exposure time With laser power level Write the resource usage record for the corresponding action so that the execution controller can issue it in sequence;

[0104] The power smoothing scheme checks the difference in laser power levels between two adjacent actions without changing the page read sequence and uses a power change threshold. A restriction is imposed when the difference in laser power level between two adjacent actions is greater than... The laser power level for the next action. Replace with the original Adjacent and with the power level difference not exceeding The power level is selected, and after the power level replacement is completed, the exposure time that meets the read error rate threshold is selected through the exposure level table, where the exposure level table is composed of LUT. Reverse construction yields and partitions are numbered for a given medium. With the replacement laser power level Iterate through the set of discrete exposure times from smallest to largest and select the one that makes the time available. The minimum exposure time as the new Then the replacement With the selected Write the resource usage plan and complete the update of the power smoothing plan.

[0105] In this specific embodiment, S6 includes:

[0106] The execution controller drives the optical read head to complete the holographic page reading and generate feedback data for reinforcement learning updates according to the page reading sequence, resource allocation plan and power smoothing plan;

[0107] Each read action in the page read sequence The process is executed sequentially, including head addressing, exposure and acquisition, buffer writing and decoding verification. For action sequence number, Number the target page node and assign it the corresponding page node attributes For batch size, For the exposure time, For the laser power level, the actuator controller sends the target angle number to the actuator at the start of the action. Target layer number With target page number And wait for the interruption to occur before sending the power level to the laser control board. And send the exposure time to the camera control panel. Camera control boards in batch sizes Continuous data acquisition The frame is written to the camera buffer block according to the resource usage plan. The decoding and error correction module performs error correction decoding on each frame and outputs error feedback and whether the reading was successful. At the same time, the buffer manager releases the buffer blocks that have completed decoding and verification in the order of buffer release to reclaim the remaining buffer capacity.

[0108] After each read operation is completed, the execution controller collects the read response time. Addressing travel distance Number of times to locate repeatedly Peak power consumption Cache blocking duration Error feedback Among them, the read response time Take the time difference from when the action enters the execution queue to when the corresponding last frame is captured, and use that as the addressing movement distance. The number of repeated positioning operations is obtained by weighting the changes in angle number, floor number, and page number according to the unit displacement coefficient specified by the equipment. The peak power consumption is obtained by counting the positioning back-off events reported by the read head controller during this action. The maximum power value sampled by the laser power supply monitoring module during this operation is obtained in watts, and the buffer blocking time is [not specified]. The cumulative time the acquisition thread waits due to the number of free blocks in the camera buffer being zero, and the error feedback read. The average bit error rate of each frame decoded within the batch is taken as the number of pages successfully read. Take the number of frames in the batch whose successful read flag is true and ;

[0109] The execution controller normalizes the above metrics to a fixed upper limit to form a penalty term and truncates it to 0 to 1, where the upper limit for read response time is... Addressing travel distance limit Maximum number of times to repeatedly locate Peak power consumption limit Maximum cache blocking time upper limit of bit error rate ;

[0110] Calculate the reward for a single action based on the benefit and penalty factors. And used to update the window policy network and page-level policy network, wherein the reward for a single action satisfies:

[0111] ;

[0112] In the formula Indicates the first The single-action reward for each reading action. This indicates reading the profit weight and taking... This indicates that the page was successfully read. Indicates batch size. Indicates the response penalty weight and takes Indicates the read response time. Indicates the maximum read response time. Indicates the weight of the move penalty and takes Indicates the addressing travel distance. Indicates the upper limit of the addressing travel distance. Indicates the relocation penalty weight and takes Indicates the number of times the location is repeated. This indicates the maximum number of times a location can be repeatedly located. Indicates the power consumption penalty weight and takes Indicates peak power consumption. Indicates the upper limit of peak power consumption. Indicates the blocking penalty weight and takes Indicates the duration of cache blocking. Indicates the maximum duration of cache blocking. Indicates the error penalty weight and takes This indicates a read error feedback. Indicates the upper limit of the bit error rate;

[0113] The training objective of the window policy network is to accumulate rewards for accessing the window, and the execution controller records the set of consecutive actions within the same access window as... The cumulative window reward is obtained by summing the rewards for each individual action. The cumulative reward for the window and the output of the window policy network that generated the access window are written together into the training cache to update the window policy network parameters. Page-level policy networks reward actions based on a single action. As a training signal and Write to the training cache to update page-level policy network parameters ;

[0114] During the online reading phase, a near-end strategy optimization algorithm is used for incremental updates with a fixed trajectory length. Collect samples and take discount factors The generalized dominance estimation coefficient is taken as The shear coefficient is taken as The optimizer uses Adam and the learning rate is... Each update involves 4 training rounds with a mini-batch size of 32. A value head is added to both the window policy network and the page-level policy network to output state values ​​and complete the advantage estimation.

[0115] During the offline training phase, training samples are constructed from historical read logs. The historical read logs record the scheduling state vector, dynamic addressing cost graph version number and edge weight, access window, page read actions and feedback indicators in time series format. The scheduling controller reconstructs the dynamic addressing cost graph and steps corresponding to each sample according to the log replay order. The addressing cost is represented by the above reward calculation rules, and the reward for each action is obtained to form a training sample set. Then, the same proximal strategy optimization algorithm as in the online reading phase is applied to... and Offline updates are performed, and the parameters completed during offline training are solidified as the initial parameters for the online reading phase. During the online reading phase, after each trajectory sample is updated, the latest parameters are written to non-volatile storage to maintain the incremental learning effect after the device restarts.

[0116] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0117] This invention unifies the spatial location of holographic pages, media partitioning status, cache usage, laser power margin, and historical error risk into state information that can be processed by the scheduling model through an angle-layer-page order heterogeneous addressing graph. This enables the hierarchical reinforcement learning strategy to update actions around response time, addressing movement distance, number of repeated positioning, peak power consumption, cache blocking, and read error feedback, thereby making the reading order and resource usage plan more suitable for high-concurrency archiving retrieval scenarios.

[0118] This invention sets up a ternary addressing cost graph dynamic update module and a safety constraint action mask module, so that the edge weights can change in real time with the angle switching time, layer movement time, page order jump distance, historical reread count, camera cache usage, laser power margin and error risk, and form a set of actionable actions before the action is executed. This improves the scheduling model's adaptability to dynamic device status and read risk, and better achieves the technical effects of reducing repeated addressing, suppressing power peaks and reducing cache blocking.

Claims

1. A holographic page read scheduling method based on reinforcement learning, characterized in that, include: S1. Obtain the concurrent archive retrieval request queue, target holographic page index, and read device status including addressing location, media partition, camera buffer, laser power, bit error rate, and readdressing records; S2. Construct a dynamic addressing cost graph based on the target holographic page index and the reading device status. The dynamic addressing cost graph uses the holographic page to be read, angle, layer, page order, and device resources as nodes, and uses reading at the same angle, layer switching, page order adjacency, resource conflict, and historical reread penalty as weighted edges. S3. Input the dynamic addressing cost graph into the graph neural network to obtain the addressing cost representation and the queue congestion representation; S4. Input the addressing cost representation and queue congestion representation into the hierarchical reinforcement learning scheduling model to generate the access window, page read order, batch size, exposure time, laser power level, and buffer release order; S5. Generate a safety constraint action mask based on the actuator's movement range, camera frame rate limit, remaining cache capacity, laser peak power, and read error rate threshold. Perform constraint correction on page read actions to obtain page read sequence, resource occupancy plan, and power smoothing plan. S6. Drive the optical read head to read holographic pages according to the page read sequence, resource usage plan and power smoothing plan, and use the read feedback to update the reward function of the hierarchical reinforcement learning scheduling model.

2. The holographic page reading scheduling method based on reinforcement learning according to claim 1, characterized in that, S1 includes: Each archive retrieval request is parsed into a request number, target hologram page number, deadline timestamp, and read priority; The target holographic page index is queried based on the target holographic page number to obtain the corresponding angle number, layer number, page sequence number and media partition number; The ratio of camera cache occupancy to cache capacity is used as the cache occupancy rate, the ratio of laser power margin to laser rated power is used as the power margin rate, the historical read error rate and repeated addressing count are written into the read risk record according to a preset sampling period, and the request number, angle number, layer number, page sequence number, media partition number, cache occupancy rate, power margin rate and read risk record are combined to form a scheduling state vector.

3. The holographic page reading scheduling method based on reinforcement learning according to claim 1, characterized in that, S2 include: Each holographic page to be read is represented as a page node. The page node includes an angle number, a layer number, a page sequence number, a media partition number, and a reading risk value. The reading risk value is obtained by weighting the historical reading error rate, the number of repeated addressings, and the media partition status according to a preset weight. Page nodes with the same angle number are connected to the continuous reading edge of the same angle; page nodes with a layer number difference equal to the preset layer step size are connected to the adjacent layer switching edge; page nodes with a page sequence number difference equal to one are connected to the page sequence adjacent edge; page nodes that share at least one of the resources in the camera buffer and laser power channel are connected to the resource conflict edge; and page nodes with historical reread records are connected to the historical reread penalty edge. The angle switching time, layer movement time, page order jump distance, cache utilization rate, power margin rate, and read risk value are normalized and weighted to obtain the addressing cost feature of the corresponding weighted edge. The addressing cost feature is then mapped monotonically to obtain the edge weight of the corresponding weighted edge.

4. The holographic page reading scheduling method based on reinforcement learning according to claim 1, characterized in that, S3 include: Embed node types, node state vectors, and edge weights into a message passing network for the input graph; In each round of message passing, the adjacent node messages are multiplied by the corresponding edge weights according to the edge type and summed. The summation result is then concatenated with the node's own state and the node update vector is obtained through the parameter matrix and nonlinear function. The page node update vector after a preset number of rounds is used as the addressing cost representation, and the queue congestion representation is obtained by concatenating the current queue length, the number of edges connecting the same resource, and the cache occupancy rate.

5. The holographic page reading scheduling method based on reinforcement learning according to claim 1, characterized in that, S4 include: The hierarchical reinforcement learning scheduling model includes a window policy network and a page-level policy network; The window policy network takes the addressing cost representation, the queue congestion representation, and the reading device status as inputs, and outputs the access window defined by the angle start and end number, the layer start and end number, and the page sequence start and end number; The page-level policy network takes the page node representation and resource node status within the access window as input, and outputs the page reading order, the batch size, the exposure time, the laser power level, and the cache release order. The batch size, the exposure time, and the laser power level are all selected from a preset discrete action table.

6. The holographic page reading scheduling method based on reinforcement learning according to claim 1, characterized in that, S5 include: A set of possible actions is generated based on the actuator's movement range, the camera's frame rate limit, the remaining buffer capacity, the laser peak power, and the read error rate threshold. Set the selection probability of page read actions that do not belong to the set of possible actions to zero; When there is a page reading action that is set to zero in the page reading order, the page reading action that is set to zero is replaced by the action with the action that is ranked first in terms of action distance. The action distance is obtained by weighted summation of angle number difference, layer number difference, page order number difference, exposure time level difference and laser power level difference. The page read sequence, the resource occupancy plan, and the power smoothing plan are generated based on the replaced page read actions.

7. The holographic page reading scheduling method based on reinforcement learning according to claim 5, characterized in that, S6 include: Read response time, addressing movement distance, number of repeated positioning, peak power consumption, cache blocking time, and read error feedback are normalized into response penalty, movement penalty, relocation penalty, power consumption penalty, blocking penalty, and error penalty, respectively. The reward for a single action is obtained by subtracting the weighted sum of all penalties from the reward item corresponding to the number of pages successfully read. The parameters of the window policy network and page-level policy network are updated based on the reward for each single action.

8. The holographic page reading scheduling method based on reinforcement learning according to claim 3, characterized in that, Update the dynamic addressing cost graph after each read operation is completed; The angle switching time is obtained by querying the actuator calibration timetable from the current read head angle to the target angle; The layer movement time is calculated from the displacement of the current layer to the target layer and the layer drive calibration speed; The page jump distance is obtained by the difference between the current page number and the target page number; The updated historical reread count, cache utilization, power margin, and read risk value are used to recalculate the addressing cost features of the corresponding weighted edges. The addressing cost features are then mapped in a monotonically decreasing manner to obtain the recalculated edge weights. The recalculated edge weights are then written into the dynamic addressing cost graph with a version number.

9. The holographic page reading scheduling method based on reinforcement learning according to claim 5, characterized in that, During the offline training phase, training samples are constructed based on historical read logs. The training samples include scheduling state vectors, dynamic addressing cost graphs, access windows, page read actions, and single action rewards. The cumulative reward for accessing the window is used as the training objective of the window policy network, and the single action reward for page reading is used as the training objective of the page-level policy network. The parameters of the window policy network and the page-level policy network are updated accordingly. After training is completed, the updated parameters are used as the initial parameters for the online reading phase, and incremental updates are performed based on the single action reward obtained in step S6 during the online reading phase.

10. The holographic page reading scheduling method based on reinforcement learning according to claim 6, characterized in that, Generating the power smoothing plan includes: The power change is calculated based on the difference in laser power levels between two consecutive reading actions; When the power change exceeds the preset power change threshold, without changing the page reading sequence, the laser power level of the next reading action is replaced with an adjacent power level that makes the power change not exceed the preset power change threshold, and the exposure time that meets the reading error rate threshold is selected according to the exposure level table. Write the replaced laser power level and selected exposure time into the resource usage plan.

Images