Latent variable dynamic supply method and related products
By using a multi-layer loading path mechanism, the dynamic supply of latent variables solves the problem of incomplete coverage of pre-calculated encoding results, improves the efficiency and continuity of robot training, and ensures the stability of model training.
Patent Information
- Application Number
- CN202611215504.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-12
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-08-12
Smart Images

Figure CN122759207B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of machine learning systems engineering and embodied intelligence, specifically to a method for dynamic supply of latent variables and related products. Background Technology
[0002] Visual-language-action models typically require multiple frames of video captured during robot task execution during training. To convert the raw video frames into data suitable for model training, a video variational autoencoder can be used to encode the original video frame sequence, obtaining corresponding latent variables, which can then be used as input data for model training.
[0003] In related technologies, encoding of the corresponding raw video frame sequences can be performed during training based on the currently required data. Since video encoding consumes computational resources and introduces processing latency, the preparation of training data may affect model training efficiency when the training data is large or the same robot task trajectory is sampled multiple times.
[0004] To reduce encoding processing during training, the original video frame sequences can be pre-encoded, and the pre-generated encoding results can be read during training. However, preprocessing of large-scale robot training data typically takes a long time, and the dataset used for training may continue to change. Therefore, the pre-generated encoding results may not cover all the original video frame ranges actually needed during training.
[0005] Furthermore, pre-generated encoding results may not be available in a timely manner due to changes in their generation or access status. In this case, continuously waiting for the encoding results may increase the preparation time for training data; stopping the corresponding data processing when the encoding results cannot be obtained may affect the continuous progress of the training process. Summary of the Invention
[0006] Embodiments of this application propose a method, apparatus, electronic device, and storage medium for the dynamic supply of latent variables.
[0007] Firstly, this application provides a method for dynamically supplying latent variables, including: Obtain the current training data request, which includes the trajectory unit identifier and the target original video frame range; Based on the trajectory unit identifier and the target original video frame interval, the segmented pre-calculated latent variable corresponding to the target original video frame interval is read through the first layer loading path, and when the segmented pre-calculated latent variable is successfully read within the first reading time budget, the segmented pre-calculated latent variable is determined as the target latent variable; When the first recoverable failure or first read timeout occurs in the first layer loading path, switch to the second layer loading path, read the long interval pre-calculated latent variable covering the target original video frame interval through the second layer loading path, and when the long interval pre-calculated latent variable is successfully read within the second read time budget, perform latent variable time dimension slicing on the long interval pre-calculated latent variable to obtain the target latent variable; When the second layer loading path experiences a second recoverable failure or a second read timeout, the system switches to the third layer loading path, reads the original video frame sequence corresponding to the target original video frame interval, and performs online encoding on the original video frame sequence using the currently trained video variational autoencoder to obtain online encoding latent variables, and determines the online encoding latent variables as the target latent variables. The target latent variable is provided to the current training step.
[0008] In some optional implementations, the step of reading the segmented pre-calculated latent variables corresponding to the target original video frame interval through the first-layer loading path based on the trajectory unit identifier and the target original video frame interval includes: Obtain the first-level availability index, which is used to characterize the correspondence between the generated segmented pre-computed latent variables and the trajectory unit identifier, the effective original video frame interval, and the availability status of the segmented pre-computed latent variables; Based on the trajectory unit identifier and the target original video frame interval, query the first layer availability index to determine the segmented pre-calculated latent variables whose start and end boundaries of the valid original video frame interval are the same as the start and end boundaries of the target original video frame interval; When the first-level availability index indicates that the corresponding segment pre-calculated latent variable is in an available state, the corresponding segment pre-calculated latent variable is read through the first-level loading path.
[0009] In some optional implementations, the step of reading long-range pre-computed latent variables covering the target original video frame range through the second-layer loading path, and performing latent variable time-dimensional slicing on the long-range pre-computed latent variables to obtain the target latent variables when the long-range pre-computed latent variables are successfully read within the second reading time budget, includes: Obtain the second-level availability index, which is used to characterize the correspondence between the generated long-interval pre-computed latent variables and the trajectory unit identifier, the effective original video frame interval, and the availability status of the long-interval pre-computed latent variables; Based on the trajectory unit identifier and the target original video frame interval, the second-level availability index is queried to determine the long interval pre-calculated latent variable that covers the target original video frame interval. The long interval pre-calculated latent variable is either a pre-calculated latent variable corresponding to the complete robot task trajectory data unit or a pre-calculated latent variable corresponding to a longer continuous original video frame interval that covers the target original video frame interval and has a coverage range greater than the target original video frame interval. When the second-level availability index indicates that the corresponding long-interval pre-calculated latent variable is in an available state, the corresponding long-interval pre-calculated latent variable is read through the second-level loading path; When the corresponding long interval pre-calculated latent variable is successfully read within the second reading time budget, the frame interval offset of the target original video frame interval relative to the starting point of the effective original video frame interval corresponding to the long interval pre-calculated latent variable is determined. Based on the frame interval offset, determine the latent variable time index range corresponding to the target original video frame interval; According to the time index range of the latent variables, the target latent variable is extracted from the long interval pre-calculated latent variables.
[0010] In some optional implementations, determining the latent variable time index range corresponding to the target original video frame interval based on the frame interval offset includes: Obtain the latent variable time grid corresponding to the long interval pre-calculated latent variable, and the latent variable time grid is used to characterize the correspondence between each latent variable time step in the long interval pre-calculated latent variable and the original video frame range; Based on the frame interval offset, determine the latent variable time step boundaries corresponding to the start and end boundaries of the target original video frame interval in the latent variable time grid, respectively. The start latent variable time index and the end latent variable time index are determined based on the latent variable time step boundaries corresponding to the start boundary and the end boundary, respectively. Based on the starting latent variable time index and the ending latent variable time index, the range of latent variable time indexes corresponding to the target original video frame interval is determined.
[0011] In some optional implementations, the current training data request further includes the target camera set and the target camera channel order; providing the target latent variables to the current training step includes: Obtain the camera channel information corresponding to the target latent variable obtained through the currently successfully loaded path; Based on the camera channel information, determine the camera channel corresponding to each target camera in the target camera set in the target latent variable; Select the latent variables of the camera channels corresponding to each target camera from the target latent variables, and arrange the selected latent variables according to the target camera channel order; The sorted latent variables are provided as the target latent variables to the current training step.
[0012] In some optional implementations, the current training data request may further include encoding configuration information, which includes a video variational autoencoder version, a temporal downsampling factor, and a preprocessing version.
[0013] The first recoverable failure includes at least one of the following: There are no segmented pre-calculated latent variables corresponding to the trajectory unit identifier and the target original video frame interval, or the corresponding segmented pre-calculated latent variables are in an unavailable state; The video variational autoencoder version, temporal downsampling factor, or preprocessing version corresponding to the pre-calculated latent variables in the corresponding segments are inconsistent with the video variational autoencoder version, temporal downsampling factor, or preprocessing version corresponding to the current training data request. Temporary network access exceptions, object storage rate limiting, or other temporary access exceptions occurred when reading the corresponding segmented pre-calculated latent variables.
[0014] The second recoverable failure includes at least one of the following: There is no long interval pre-calculated latent variable that corresponds to the trajectory unit identifier and whose valid original video frame interval covers the target original video frame interval, or the corresponding long interval pre-calculated latent variable is in an unavailable state; The video variational autoencoder version, temporal downsampling factor, or preprocessing version corresponding to the pre-calculated latent variables over the corresponding long interval is inconsistent with the video variational autoencoder version, temporal downsampling factor, or preprocessing version corresponding to the current training data request. It is impossible to determine the latent variable time index range that can be used to perform latent variable time dimension slicing on the corresponding long interval based on the target original video frame interval; Temporary network access exceptions, object storage rate limiting, or other temporary access exceptions may occur when reading the corresponding long interval pre-calculated latent variables.
[0015] In some optional implementations, before determining the segmented pre-computed latent variables obtained through the first-layer loading path as the target latent variables, or before performing latent variable time dimension slicing on the long-interval pre-computed latent variables obtained through the second-layer loading path, the method further includes: Determine the request consistency field corresponding to the current training data request. The request consistency field includes at least one of the following configuration fields: trajectory unit identifier, target range, camera configuration information, encoding configuration information, video preprocessing information, and latent variable format information. The target range includes the target original video frame interval or its corresponding target latent variable time range. Extract the segmented consistency field corresponding to the request consistency field from the metadata of the segmented pre-computed latent variables, or determine the long interval consistency field corresponding to the part of the latent variables to be cut out based on the metadata of the long interval pre-computed latent variables. Following the same field order and deterministic serialization rules, a request configuration fingerprint is generated based on the request consistency field, and a segmented configuration fingerprint or a long interval configuration fingerprint is generated based on the segmented consistency field or the long interval consistency field. The request configuration fingerprint is compared with the segmented configuration fingerprint or the long interval configuration fingerprint, and target range coverage verification and data integrity verification are performed on the segmented pre-calculated latent variables or the long interval pre-calculated latent variables; When the configured fingerprint is consistent, covers the target range, and passes the data integrity verification, it is determined that the segmented pre-calculated latent variable meets the usage conditions as the target latent variable, or it is determined that the long interval pre-calculated latent variable meets the usage conditions for executing latent variable time dimension slicing. When the configured fingerprint is inconsistent or does not cover the target range, the first recoverable failure is determined to occur in the first layer loading path based on the segmented pre-calculated latent variables, or the second recoverable failure is determined to occur in the second layer loading path based on the long interval pre-calculated latent variables. When the segmented pre-calculated latent variable fails the data integrity check, the data integrity anomaly of the segmented pre-calculated latent variable is recorded and isolated, and the process is switched to the second-level loading path; when the long-interval pre-calculated latent variable fails the data integrity check, the data integrity anomaly of the long-interval pre-calculated latent variable is recorded and isolated, and the process is switched to the third-level loading path.
[0016] In some optional implementations, when reading the segmented pre-calculated latent variables corresponding to the target original video frame interval through the first-level loading path, the method further includes: The moment when the actual reading of the segmented pre-calculated latent variables begins is determined as the first reading start moment; The time elapsed from the first reading start time to the first reading time budget is defined as the first budget cutoff time. When the segmented pre-calculated latent variable is read no later than the first budget deadline, it is determined that the segmented pre-calculated latent variable has been successfully read within the first reading time budget; when the segmented pre-calculated latent variable has not been read by the first budget deadline, it is determined that the first reading timeout has occurred.
[0017] When reading long-interval pre-calculated latent variables covering the target original video frame interval through the second-layer loading path, the method further includes: The moment when the actual reading of the long interval pre-calculated latent variables is determined as the second reading start time; The time elapsed from the second reading start time to the second reading time budget is determined as the second budget cutoff time. When the long-interval pre-calculated latent variable is read no later than the second budget deadline, it is determined that the long-interval pre-calculated latent variable has been successfully read within the second reading time budget; when the long-interval pre-calculated latent variable has not been read by the second budget deadline, it is determined that the second reading timeout has occurred.
[0018] In some alternative implementations, the method further includes, before starting to read the segmented pre-computed latent variables and before starting to read the long-interval pre-computed latent variables: Obtain a first delay sample set of the first layer loading path within a preset statistical window, and a second delay sample set of the second layer loading path within the preset statistical window. The first delay sample set and the second delay sample set respectively include the read delay of the corresponding loading path that was recently successfully read. Determine the first historical latency quantile value of the first latency sample set, and determine the first candidate time budget based on the first historical latency quantile value and the first time margin reserved for reading latency fluctuations; The first candidate time budget is limited to between a preset first minimum time budget and a preset first maximum time budget to obtain the first reading time budget; Determine the second historical latency quantile value of the second latency sample set, and determine the second candidate time budget based on the second historical latency quantile value and the second time margin reserved for reading latency fluctuations; The second candidate time budget is limited to between a preset second minimum time budget and a preset second maximum time budget to obtain the second reading time budget.
[0019] In some optional implementations, after determining the online encoded latent variable as the target latent variable, the method further includes: Within a preset statistical window or since the start of training, the cumulative number of online encodings of robot task trajectory data units with the trajectory unit identifier entering the third-layer loading path; Asynchronous preheating submission judgment is performed only after the number of online encoding attempts reaches a preset online encoding attempt threshold; When the result of the asynchronous preheating submission judgment is met, and the preheating task submission conditions are met, after providing the target latent variable to the current training step, an asynchronous background pre-computation task is submitted to generate the corresponding segmented pre-computation latent variable or long interval pre-computation latent variable.
[0020] In some optional implementations, the asynchronous preheating submission judgment includes: Based on the sampling frequency of the corresponding robot task trajectory data unit and the expected number of remaining training steps, the expected number of visits to the robot task trajectory data unit in the remaining training process is determined. The expected net benefit of performing pre-computation is determined based on the estimated number of accesses, the single-pass latency savings between online encoding and pre-computed latent variable readings, and the equivalent cost required to perform pre-computation. If the expected net income meets the preset income conditions, the preheating task submission conditions are determined to be met; if the expected net income does not meet the preset income conditions, the preheating task submission conditions are determined not to be met.
[0021] When multiple submitted background pre-computation tasks exist, the method further includes: The scheduling priority of each background pre-calculation task is determined based on the sampling frequency, expected delay saving per cycle, pre-calculation success probability, and pre-calculation equivalent cost of the robot task trajectory data unit corresponding to each background pre-calculation task. The execution order of the multiple background pre-computation tasks is determined according to the scheduling priority from high to low.
[0022] Secondly, this application provides a latent variable dynamic supply device, comprising: The request acquisition module is used to acquire the current training data request, which includes the trajectory unit identifier and the target original video frame range; The first-layer loading module is used to read the segmented pre-calculated latent variables corresponding to the target original video frame interval through the first-layer loading path according to the trajectory unit identifier and the target original video frame interval, and when the segmented pre-calculated latent variables are successfully read within the first reading time budget, the segmented pre-calculated latent variables are determined as target latent variables. The second-layer loading module is used to read long-range pre-calculated latent variables covering the target original video frame range through the second-layer loading path when the first recoverable failure or the first reading timeout occurs in the first-layer loading path, and to perform latent variable time dimension slicing on the long-range pre-calculated latent variables to obtain the target latent variables when the long-range pre-calculated latent variables are successfully read within the second reading time budget. The third-layer loading module is used to read the original video frame sequence corresponding to the target original video frame interval when the second-layer loading path experiences a second recoverable failure or a second read timeout, and to perform online encoding on the original video frame sequence using the currently trained video variational autoencoder to obtain online encoding latent variables, and to determine the online encoding latent variables as the target latent variables. The latent variable providing module is used to provide the target latent variable to the current training step.
[0023] In some optional implementations, the first layer loading module is further configured to: Obtain the first-level availability index, which is used to characterize the correspondence between the generated segmented pre-computed latent variables and the trajectory unit identifier, the effective original video frame interval, and the availability status of the segmented pre-computed latent variables; Based on the trajectory unit identifier and the target original video frame interval, query the first layer availability index to determine the segmented pre-calculated latent variables whose start and end boundaries of the valid original video frame interval are the same as the start and end boundaries of the target original video frame interval; When the first-level availability index indicates that the corresponding segment pre-calculated latent variable is in an available state, the corresponding segment pre-calculated latent variable is read through the first-level loading path.
[0024] In some optional implementations, the second-layer loading module is further configured to: Obtain the second-level availability index, which is used to characterize the correspondence between the generated long-interval pre-computed latent variables and the trajectory unit identifier, the effective original video frame interval, and the availability status of the long-interval pre-computed latent variables; Based on the trajectory unit identifier and the target original video frame interval, the second-level availability index is queried to determine the long interval pre-calculated latent variable that covers the target original video frame interval. The long interval pre-calculated latent variable is either a pre-calculated latent variable corresponding to the complete robot task trajectory data unit or a pre-calculated latent variable corresponding to a longer continuous original video frame interval that covers the target original video frame interval and has a coverage range greater than the target original video frame interval. When the second-level availability index indicates that the corresponding long-interval pre-calculated latent variable is in an available state, the corresponding long-interval pre-calculated latent variable is read through the second-level loading path; When the corresponding long interval pre-calculated latent variable is successfully read within the second reading time budget, the frame interval offset of the target original video frame interval relative to the starting point of the effective original video frame interval corresponding to the long interval pre-calculated latent variable is determined. Based on the frame interval offset, determine the latent variable time index range corresponding to the target original video frame interval; According to the time index range of the latent variables, the target latent variable is extracted from the long interval pre-calculated latent variables.
[0025] In some optional implementations, when determining the latent variable time index range corresponding to the target original video frame interval based on the frame interval offset, the second-layer loading module is further configured to: Obtain the latent variable time grid corresponding to the long interval pre-calculated latent variable, and the latent variable time grid is used to characterize the correspondence between each latent variable time step in the long interval pre-calculated latent variable and the original video frame range; Based on the frame interval offset, determine the latent variable time step boundaries corresponding to the start and end boundaries of the target original video frame interval in the latent variable time grid, respectively. The start latent variable time index and the end latent variable time index are determined based on the latent variable time step boundaries corresponding to the start boundary and the end boundary, respectively. Based on the starting latent variable time index and the ending latent variable time index, the range of latent variable time indexes corresponding to the target original video frame interval is determined.
[0026] In some optional implementations, the current training data request may further include the target camera set and the target camera channel order; When providing the target latent variable to the current training step, the latent variable providing module is further configured to: Obtain the camera channel information corresponding to the target latent variable obtained through the currently successfully loaded path; Based on the camera channel information, determine the camera channel corresponding to each target camera in the target camera set in the target latent variable; Select the latent variables of the camera channels corresponding to each target camera from the target latent variables, and arrange the selected latent variables according to the target camera channel order; The sorted latent variables are provided as the target latent variables to the current training step.
[0027] In some optional implementations, the current training data request may further include encoding configuration information, which includes a video variational autoencoder version, a temporal downsampling factor, and a preprocessing version. The first recoverable failure includes at least one of the following: There are no segmented pre-calculated latent variables corresponding to the trajectory unit identifier and the target original video frame interval, or the corresponding segmented pre-calculated latent variables are in an unavailable state; The video variational autoencoder version, temporal downsampling factor, or preprocessing version corresponding to the pre-calculated latent variables in the corresponding segments are inconsistent with the video variational autoencoder version, temporal downsampling factor, or preprocessing version corresponding to the current training data request. A temporary network access exception, object storage rate limiting, or other temporary access exception occurred when reading the corresponding segmented pre-calculated latent variables. The second recoverable failure includes at least one of the following: There is no long interval pre-calculated latent variable that corresponds to the trajectory unit identifier and whose valid original video frame interval covers the target original video frame interval, or the corresponding long interval pre-calculated latent variable is in an unavailable state; The video variational autoencoder version, temporal downsampling factor, or preprocessing version corresponding to the pre-calculated latent variables over the corresponding long interval is inconsistent with the video variational autoencoder version, temporal downsampling factor, or preprocessing version corresponding to the current training data request. It is impossible to determine the latent variable time index range that can be used to perform latent variable time dimension slicing on the corresponding long interval based on the target original video frame interval; Temporary network access exceptions, object storage rate limiting, or other temporary access exceptions may occur when reading the corresponding long interval pre-calculated latent variables.
[0028] In some optional implementations, before determining the segmented pre-computed latent variables obtained through the first-layer loading path as the target latent variables, or before performing latent variable time dimension slicing on the long-interval pre-computed latent variables obtained through the second-layer loading path, the first-layer loading module or the second-layer loading module is further configured to: Determine the request consistency field corresponding to the current training data request. The request consistency field includes at least one of the following configuration fields: trajectory unit identifier, target range, camera configuration information, encoding configuration information, video preprocessing information, and latent variable format information. The target range includes the target original video frame interval or its corresponding target latent variable time range. Extract the segmented consistency field corresponding to the request consistency field from the metadata of the segmented pre-computed latent variables, or determine the long interval consistency field corresponding to the part of the latent variables to be cut out based on the metadata of the long interval pre-computed latent variables. Following the same field order and deterministic serialization rules, a request configuration fingerprint is generated based on the request consistency field, and a segmented configuration fingerprint or a long interval configuration fingerprint is generated based on the segmented consistency field or the long interval consistency field. The request configuration fingerprint is compared with the segmented configuration fingerprint or the long interval configuration fingerprint, and target range coverage verification and data integrity verification are performed on the segmented pre-calculated latent variables or the long interval pre-calculated latent variables; When the configured fingerprint is consistent, covers the target range, and passes the data integrity verification, it is determined that the segmented pre-calculated latent variable meets the usage conditions as the target latent variable, or it is determined that the long interval pre-calculated latent variable meets the usage conditions for executing latent variable time dimension slicing. When the configured fingerprint is inconsistent or does not cover the target range, the first recoverable failure is determined to occur in the first layer loading path based on the segmented pre-calculated latent variables, or the second recoverable failure is determined to occur in the second layer loading path based on the long interval pre-calculated latent variables. When the segmented pre-calculated latent variable fails the data integrity check, the data integrity anomaly of the segmented pre-calculated latent variable is recorded and isolated, and the process is switched to the second-level loading path; when the long-interval pre-calculated latent variable fails the data integrity check, the data integrity anomaly of the long-interval pre-calculated latent variable is recorded and isolated, and the process is switched to the third-level loading path.
[0029] In some optional implementations, when the first-layer loading module reads the segmented pre-calculated latent variables corresponding to the target original video frame interval through the first-layer loading path, it is further used to: The moment when the actual reading of the segmented pre-calculated latent variables begins is determined as the first reading start moment; The time elapsed from the first reading start time to the first reading time budget is defined as the first budget cutoff time. When the segmented pre-calculated latent variable is read no later than the first budget deadline, it is determined that the segmented pre-calculated latent variable has been successfully read within the first reading time budget; when the segmented pre-calculated latent variable has not been read by the first budget deadline, it is determined that the first reading timeout has occurred.
[0030] When the second-layer loading module reads the long-interval pre-calculated latent variables covering the target original video frame interval through the second-layer loading path, it is further used for: The moment when the actual reading of the long interval pre-calculated latent variables is determined as the second reading start time; The time elapsed from the second reading start time to the second reading time budget is determined as the second budget cutoff time. When the long-interval pre-calculated latent variable is read no later than the second budget deadline, it is determined that the long-interval pre-calculated latent variable has been successfully read within the second reading time budget; when the long-interval pre-calculated latent variable has not been read by the second budget deadline, it is determined that the second reading timeout has occurred.
[0031] In some optional implementations, before starting to read the segmented pre-computed latent variables and before starting to read the long-interval pre-computed latent variables, the first-layer loading module and the second-layer loading module are further configured to: Obtain a first delay sample set of the first layer loading path within a preset statistical window, and a second delay sample set of the second layer loading path within the preset statistical window. The first delay sample set and the second delay sample set respectively include the read delay of the corresponding loading path that was recently successfully read. Determine the first historical latency quantile value of the first latency sample set, and determine the first candidate time budget based on the first historical latency quantile value and the first time margin reserved for reading latency fluctuations; The first candidate time budget is limited to between a preset first minimum time budget and a preset first maximum time budget to obtain the first reading time budget; Determine the second historical latency quantile value of the second latency sample set, and determine the second candidate time budget based on the second historical latency quantile value and the second time margin reserved for reading latency fluctuations; The second candidate time budget is limited to between a preset second minimum time budget and a preset second maximum time budget to obtain the second reading time budget.
[0032] In some alternative implementations, the apparatus further includes an asynchronous preheating module.
[0033] After the third-layer loading module determines the online encoded latent variable as the target latent variable, the asynchronous preheating module is used to: Within a preset statistical window or since the start of training, the cumulative number of online encodings of robot task trajectory data units with the trajectory unit identifier entering the third-layer loading path; Asynchronous preheating submission judgment is performed only after the number of online encoding attempts reaches a preset online encoding attempt threshold; When the result of the asynchronous preheating submission judgment is met, and the preheating task submission conditions are met, after providing the target latent variable to the current training step, an asynchronous background pre-computation task is submitted to generate the corresponding segmented pre-computation latent variable or long interval pre-computation latent variable.
[0034] In some optional implementations, the asynchronous preheating module, when performing the asynchronous preheating submission judgment, is further configured to: Based on the sampling frequency of the corresponding robot task trajectory data unit and the expected number of remaining training steps, the expected number of visits to the robot task trajectory data unit in the remaining training process is determined. The expected net benefit of performing pre-computation is determined based on the estimated number of accesses, the single-pass latency savings between online encoding and pre-computed latent variable readings, and the equivalent cost required to perform pre-computation. If the expected net income meets the preset income conditions, the preheating task submission conditions are determined to be met; if the expected net income does not meet the preset income conditions, the preheating task submission conditions are determined not to be met.
[0035] When multiple submitted background pre-computation tasks exist, the asynchronous preheating module is further used to: The scheduling priority of each background pre-calculation task is determined based on the sampling frequency, expected delay saving per cycle, pre-calculation success probability, and pre-calculation equivalent cost of the robot task trajectory data unit corresponding to each background pre-calculation task. The execution order of the multiple background pre-computation tasks is determined according to the scheduling priority from high to low.
[0036] Thirdly, this application provides an electronic device, including: one or more processors; a storage device having one or more programs stored thereon; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method described in any embodiment of the first aspect of this application.
[0037] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the first aspect of this application.
[0038] Fifthly, this application provides a computer program product, including a computer program or computer instructions, which, when executed by a processor, implement the method described in any embodiment of the first aspect of this application.
[0039] This application prioritizes reading segmented pre-computed latent variables corresponding to the target original video frame interval by sequentially setting first-layer, second-layer, and third-layer loading paths. This reduces the computational overhead and processing latency caused by online encoding of the original video frame sequence. When the segmented pre-computed latent variables corresponding to the target original video frame interval cannot be successfully read, this application reads long-interval pre-computed latent variables covering the target original video frame interval and obtains the target latent variable through latent variable time dimension slicing. Thus, even when the pre-generated encoding results cannot directly cover the currently required video frame interval, the application utilizes existing pre-computed latent variables with a wider coverage to obtain the target latent variable required for the current training step, improving the utilization rate of existing pre-computed latent variables. When segmented pre-computed latent variables or long-interval pre-computed latent variables cannot be successfully read within the corresponding reading time budget, this application switches to the next layer loading path to avoid continuously waiting for pre-computed latent variables that cannot be returned in time. When neither segmented pre-computed latent variables nor long-interval pre-computed latent variables can be successfully used, the target latent variables required for the current training step are obtained and provided by performing online encoding on the original video frame sequence corresponding to the target original video frame interval, thereby ensuring the continuity of training data supply.
[0040] Therefore, this application can prioritize the use of existing pre-computed latent variables to reduce the cost of training data preparation when the pre-generated latent variables are incomplete or cannot be obtained in a timely manner, and continuously obtain the target latent variables through a layer-by-layer switching loading method, thereby balancing the efficiency of training data preparation for the visual-language-action model and the continuity of the training process. Attached Figure Description
[0041] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the scope of this application. In the drawings: Figure 1 This is a system architecture diagram of one embodiment of the latent variable dynamic supply method according to this application; Figure 2 This is a flowchart of an embodiment of the latent variable dynamic supply method according to this application; Figure 3 This is an exploded flowchart of an embodiment of step S203 of this application; Figure 4 This is a flowchart of an embodiment of performing asynchronous preheating processing according to this application; Figure 5 This is a schematic diagram of a structure of an embodiment of the latent variable dynamic supply device according to this application; Figure 6This is a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of this application. Detailed Implementation
[0042] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.
[0043] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0044] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of the latent variable dynamic supply method, apparatus, electronic device, and storage medium of this application can be applied.
[0045] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 provides a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired communication links, wireless communication links, or fiber optic communication links.
[0046] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send data. Terminal devices 101, 102, and 103 can have applications for visual-language-action model training, training data loading, data management, or other applications related to model training installed.
[0047] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be desktop computers, portable computers, workstations, training servers, or other electronic devices with data processing capabilities. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. Terminal devices 101, 102, and 103 can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here.
[0048] Server 105 can be a server that provides various services, such as a backend server that processes current training data requests sent by terminal devices 101, 102, and 103. The backend server can obtain the target latent variable based on the current training data request through a first-level loading path, a second-level loading path, or a third-level loading path, and provide the target latent variable to the current training step.
[0049] In some cases, the latent variable dynamic supply method provided in this application can be jointly executed by terminal devices 101, 102, and 103 and server 105. For example, the processing of obtaining the current training data request and providing the target latent variable to the current training step can be executed by terminal devices 101, 102, and 103, while the processing of obtaining the target latent variable through the first-layer loading path, the second-layer loading path, or the third-layer loading path can be executed by server 105. This application does not limit this. Correspondingly, the latent variable dynamic supply device can also be respectively set in terminal devices 101, 102, and 103 and server 105.
[0050] In some cases, the latent variable dynamic supply method provided in this application can be executed by server 105, and correspondingly, the latent variable dynamic supply device can also be set in server 105. In this case, system architecture 100 may not include terminal devices 101, 102, and 103.
[0051] In some cases, the latent variable dynamic supply method provided in this application can be executed by terminal devices 101, 102, and 103, and correspondingly, the latent variable dynamic supply device can also be set in terminal devices 101, 102, and 103. In this case, system architecture 100 may not include server 105.
[0052] Server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here.
[0053] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown in the diagram is for illustrative purposes only. Depending on the specific implementation requirements, system architecture 100 may include any number of terminal devices, networks, and servers.
[0054] Continue to refer to Figure 2 , Figure 2 A flowchart of one embodiment of the latent variable dynamic supply method according to this application is shown. Figure 2 The method shown can be derived from Figure 1The process is executed by the terminal device or server in the process. This process 200 includes the following steps S201 to S205.
[0055] Step S201: Obtain the current training data request.
[0056] The current training data request includes trajectory unit identifiers and target raw video frame ranges.
[0057] Specifically, the current training data request is a data request generated by the current training step to obtain the target latent variables required for the current model computation. The current training step can be a training step in the training process of a vision-language-action model.
[0058] A robot task trajectory data unit is a data unit formed during the execution of a robot's task or a period of operation. A robot task trajectory data unit may include a sequence of raw video frames captured by one or more cameras, as well as robot motion sequences, robot state information, task attribute information, time information, or other information related to the robot's task process.
[0059] Trajectory unit identifiers are used to distinguish different robot task trajectory data units. They can be internal identifiers of the dataset, object storage identifiers, universally unique identifiers, or identifiers generated based on the attributes of the robot task trajectory data units.
[0060] The target original video frame interval represents the range of original video frames required for the current training step. The target original video frame interval can be represented by a start boundary and an end boundary.
[0061] In addition, the same interval boundary rules can be used for the current training data request, as well as the metadata for subsequent availability indexes and pre-computed latent variables.
[0062] For example, a left-closed, right-open interval can be used, where the starting boundary includes the frame but the ending boundary does not, to avoid adjacent intervals repeatedly containing the same original video frame at the ending boundary.
[0063] In some implementations, the current training data request may also include the target latent variable time range. The target latent variable time range and the target original video frame interval can be converted to each other according to the temporal downsampling relationship of the video variational autoencoder.
[0064] The current training data request may also include a training task identifier, a dataset identifier, a training process identifier, or other information used to determine the purpose of latent variables.
[0065] In some alternative implementations, the current training data request may also include the target camera set and the target camera channel order.
[0066] Specifically, a robot task trajectory data unit may include a sequence of raw video frames from multiple cameras.
[0067] For example, cameras can be installed on the robot's head, left robotic arm, and right robotic arm respectively.
[0068] The target camera set is used to indicate which latent variables corresponding to which cameras should be used in the current training step.
[0069] The target camera channel order is used to indicate the order in which the latent variables corresponding to each target camera are provided to the current training step.
[0070] The camera channel order can be understood as the index order of the camera dimension in the target latent variables. It is used to avoid camera view mismatch caused by different loading paths organizing latent variables according to different camera orders.
[0071] In some optional implementations, the current training data request may also include encoding configuration information, which includes the video variational autoencoder version, the temporal downsampling factor, and the preprocessing version.
[0072] Specifically, the video variational autoencoder version identifies the video variational autoencoder used to encode the original video frame sequence. Different video variational autoencoder versions can correspond to different model structures, model parameters, or encoding implementations. The main purpose of the video variational autoencoder version is to determine whether the video variational autoencoder used for pre-computing latent variables is consistent with the video variational autoencoder used in the current training, so as to avoid providing latent variables generated based on different video variational autoencoders to the current training step.
[0073] The temporal downsampling factor represents the downsampling relationship between the temporal dimension of the original video frame sequence and the temporal dimension of the latent variables. The same temporal downsampling factor can be used for video frame sequences from different cameras within a single robot task trajectory data unit. The temporal downsampling factor is primarily used to determine whether the temporal coding relationship between the pre-computed latent variables and the current training data request is consistent, and to determine the correspondence between the original video frame intervals and the temporal index range of the latent variables.
[0074] The preprocessed version identifies the resolution adjustment, image normalization, image cropping, or other video preprocessing methods performed on the original video frame sequence. Multiple different preprocessed versions can be generated from the same original video frame sequence. The preprocessed version is primarily used to determine whether the video preprocessing method used to generate the pre-computed latent variables is consistent with the current training data request.
[0075] In some alternative implementations, both the first-layer loading path and the second-layer loading path need to read pre-computed latent variables from their respective storage locations. To avoid the current training step from waiting due to the slow reading speed of a pre-computed latent variable even though it exists, a first reading time budget can be determined for the first-layer loading path and a second reading time budget can be determined for the second-layer loading path before the actual reading begins.
[0076] Therefore, before starting to read segmented pre-computed latent variables and before starting to read long interval pre-computed latent variables, the following (A1)-(A5) may be included.
[0077] (A1) can obtain the first delay sample set of the first layer loading path within the preset statistics window, and the second delay sample set of the second layer loading path within the preset statistics window. The first delay sample set and the second delay sample set respectively include the reading delay of the recently successfully read corresponding loading path.
[0078] Specifically, the preset statistics window can be a preset duration ending at the current time, or it can be a statistics window that includes the most recent preset number of successful read operations for the corresponding loading path.
[0079] Each read delay in the first delay sample set can represent the time elapsed from the actual start of reading segmented pre-computed latent variables to the successful completion of the read in the first-level loading path.
[0080] Each read delay in the second delay sample set can represent the time elapsed from the actual start of reading the long interval pre-computed latent variables to the successful completion of the read in the second-level loading path.
[0081] (A2) Determine the first historical delay quantile value of the first delay sample set, and determine the first candidate time budget based on the first historical delay quantile value and the first time margin reserved for reading delay fluctuations.
[0082] Specifically, the read delays in the first delay sample set can be arranged in ascending order of read delay, and the first historical delay quantile value can be determined according to the preset quantile parameter.
[0083] Preset quantile parameters can be, for example, the 90th percentile, the 95th percentile, or other quantile parameters that can characterize the recent read latency distribution of the first-level loading path.
[0084] The initial buffer is used to absorb network jitter, changes in storage backend load, task scheduling jitter, or other short-term latency fluctuations.
[0085] The first candidate time budget can be determined by the sum of the first historical delay quantile and the first time margin.
[0086] (A3) The first candidate time budget is limited to between the preset first minimum time budget and the preset first maximum time budget to obtain the first reading time budget.
[0087] Specifically, when the first candidate time budget is less than the preset first minimum time budget, the preset first minimum time budget can be determined as the first reading time budget.
[0088] When the first candidate time budget is greater than the preset first maximum time budget, the preset first maximum time budget can be determined as the first reading time budget.
[0089] When the first candidate time budget is between the preset first minimum time budget and the preset first maximum time budget, the first candidate time budget can be determined as the first reading time budget.
[0090] This approach avoids both excessively short first-read time budgets that could lead to frequent timeouts in the first-layer loading path and excessively long first-read time budgets that could cause the current training step to wait for an extended period.
[0091] (A4) Determine the second historical delay quantile of the second delay sample set, and determine the second candidate time budget based on the second historical delay quantile and the second time margin reserved for reading delay fluctuations.
[0092] Specifically, the read delays in the second delay sample set can be arranged in ascending order of read delay, and the second historical delay quantile value can be determined according to the preset quantile parameter.
[0093] The second time margin is used to absorb network jitter, changes in storage backend load, task scheduling jitter, or other short-term latency fluctuations that may occur when the second-layer loading path reads long-range pre-calculated latent variables.
[0094] The second candidate time budget can be determined by the sum of the second historical delay quantile and the second time margin. The first-level loading path and the second-level loading path can use the same or different preset quantile parameters, or the same or different time margins.
[0095] (A5) The second candidate time budget is limited to between the preset second minimum time budget and the preset second maximum time budget to obtain the second reading time budget.
[0096] Specifically, when the second candidate time budget is less than the preset second minimum time budget, the preset second minimum time budget can be determined as the second reading time budget.
[0097] When the second candidate time budget is greater than the preset second maximum time budget, the preset second maximum time budget can be determined as the second reading time budget.
[0098] When the second candidate time budget is between the preset second minimum time budget and the preset second maximum time budget, the second candidate time budget can be determined as the second reading time budget.
[0099] The preset second minimum time budget and the preset second maximum time budget can be determined based on the amount of data, storage location, storage backend type, and historical read performance of the latent variables in the long interval.
[0100] The process of determining the first read time budget and the second read time budget mentioned above can be uniformly expressed as formula (1): (1) in, In this embodiment, the loading path hierarchy is indicated. It can represent the first-level loading path or the second-level loading path; Indicates the first The set of delayed samples within a preset statistical window for the layer loading path; Represents the delayed sample set The historical delay quantile value; τ represents the preset quantile parameter; Represented as the first The time margin reserved for the read latency fluctuation of the layer loading path; and They represent the first Minimum and maximum time budgets for the layer loading path; This indicates that the candidate time budget will be limited to between the minimum and maximum time budgets; Indicates the first The final read time budget for the layer loading path. All the above time quantities use the same time unit, such as milliseconds.
[0101] when When indicating the first-level loading path, , , , , and These correspond to the first latency sample set, the first historical latency quantile, the first time margin, the preset first minimum time budget, the preset first maximum time budget, and the first read time budget, respectively. When indicating the second-level loading path, the above parameters correspond to the second delay sample set, the second historical delay quantile, the second time margin, the preset second minimum time budget, the preset second maximum time budget, and the second reading time budget, respectively.
[0102] For example, the recent successful read latencies of the first-level loading path are 12 milliseconds, 13 milliseconds, 15 milliseconds, 16 milliseconds, 18 milliseconds, 20 milliseconds, 22 milliseconds, 25 milliseconds, 28 milliseconds, and 35 milliseconds, respectively. Using the 90th percentile as the first historical latency quantile, the first historical latency quantile can be 28 milliseconds. With a first time margin of 5 milliseconds, a preset first minimum time budget of 15 milliseconds, and a preset first maximum time budget of 40 milliseconds, the first candidate time budget is 33 milliseconds. Since 33 milliseconds falls between 15 milliseconds and 40 milliseconds, 33 milliseconds can be determined as the first read time budget.
[0103] When the number of read delays in the first delay sample set or the second delay sample set is lower than the preset number, the corresponding first read time budget or second read time budget is determined based on the initial default time budget of the corresponding loading path or the historical delay statistics of the corresponding storage backend.
[0104] Specifically, at the beginning of training or when the number of successful reads on the corresponding loading path is low, the read latency in the latency sample set may not be sufficient to form representative historical latency quantiles. In this case, instead of calculating the read time budget based on the corresponding latency sample set, a pre-configured initial default time budget can be used, or the corresponding read time budget can be determined based on historical latency statistics of the storage backend where segmented or long-interval pre-calculated latent variables are located. As the number of successful reads increases and reaches a preset threshold, the first or second read time budget can be dynamically updated based on the read latency of recently successful reads on the corresponding loading path.
[0105] Step S202: Based on the trajectory unit identifier and the target original video frame interval, read the segmented pre-calculated latent variables corresponding to the target original video frame interval through the first layer loading path, and when the segmented pre-calculated latent variables are successfully read within the first reading time budget, determine the segmented pre-calculated latent variables as target latent variables.
[0106] Specifically, the first-level loading path has a higher attempt priority than the second-level and third-level loading paths.
[0107] Segmented pre-computed latent variables are latent variables pre-generated for the original video frame intervals that can be directly requested during the training process. The corresponding effective original video frame intervals can be the same as the target original video frame intervals.
[0108] Therefore, when the usage conditions such as availability, configuration consistency, target scope coverage and data integrity are met, the read segmented pre-computed latent variables can be directly determined as target latent variables without performing latent variable time dimension slicing or online encoding.
[0109] In some optional implementations, based on the trajectory unit identifier and the target original video frame interval, the segmented pre-calculated latent variables corresponding to the target original video frame interval are read through the first-layer loading path, which can be achieved in the following ways.
[0110] First, the first-level availability index is obtained. The first-level availability index is used to characterize the correspondence between the generated segmented pre-computed latent variables and the trajectory unit identifier, the effective original video frame interval, and the availability status of the segmented pre-computed latent variables.
[0111] Specifically, the first-level loading path can find the segmented pre-computed latent variable through the first-level availability index, and read the segmented pre-computed latent variable when the index record indicates that the corresponding segmented pre-computed latent variable has been generated and can be read.
[0112] The first-level availability index is not the segmented pre-computed latent variable itself, but rather a data structure used to look up the segmented pre-computed latent variable.
[0113] In some alternative implementations, a first-level availability index record may include not only the trajectory unit identifier, valid raw video frame range, and availability status, but also the storage location and metadata location.
[0114] The storage location indicates the location of the corresponding segmented pre-computed latent variables in local storage, a network file system, object storage, or other storage backend. The metadata location indicates the location where the generation configuration, camera channel information, effective range, and integrity information of the segmented pre-computed latent variables are stored.
[0115] Then, based on the trajectory unit identifier and the target original video frame interval, the first-level availability index is queried to determine the segmented pre-calculated latent variables whose start and end boundaries of the valid original video frame interval are the same as the start and end boundaries of the target original video frame interval.
[0116] Specifically, in the first-level availability index, search for index records where the trajectory unit identifier is the same as the trajectory unit identifier in the current training data request, and the start and end boundaries of the valid original video frame interval are the same as the start and end boundaries of the target original video frame interval, respectively.
[0117] The term "same" here means that, when the same interval boundary rules are used, the two intervals have the same starting and ending boundaries, rather than simply requiring that the two intervals have an intersection.
[0118] For example, the trajectory unit identifier corresponding to the current training data request is A, and the target original video frame range is from frame 120 to frame 152. If an index record in the first-level availability index corresponds to trajectory unit identifier A, and the effective original video frame range is also from frame 120 to frame 152, then the segmented pre-computed latent variable corresponding to this index record can be determined as the segmented pre-computed latent variable corresponding to the target original video frame range.
[0119] Next, when the first-level availability index indicates that the corresponding segment pre-computed latent variable is in an available state, the corresponding segment pre-computed latent variable is read through the first-level loading path.
[0120] Specifically, the availability status indicates that the corresponding segmented pre-computed latent variable has been generated, stored, and written; the index record is active and the corresponding data can be read. Segmented pre-computed latent variables that are being generated, have not yet been written, have been deleted, or have been isolated due to data integrity issues are not in an availability status. Here, a read can be initiated based on the storage location in the index record, and after the read is completed, the verification process described later will determine whether the segmented pre-computed latent variable can be used in the current training step.
[0121] In one optional implementation, the segmented pre-computed latent variable corresponding to the target original video frame interval can be composed of a single segmented pre-computed latent variable or a combination of multiple adjacent segmented pre-computed latent variables. When a single segmented pre-computed latent variable cannot cover the target original video frame interval, but the union of the valid original video frame intervals of multiple adjacent segmented pre-computed latent variables is the same as the target original video frame interval, the multiple adjacent segmented pre-computed latent variables can be determined from the first-level availability index.
[0122] Before combining multiple adjacent segmented pre-computed latent variables, it can be verified that these adjacent segmented pre-computed latent variables correspond to the same trajectory unit identifier, have consistent camera configuration information, encoding configuration information, video preprocessing information, and latent variable format information, and that there are no gaps or overlaps between adjacent valid original video frame intervals. It can also be verified that the boundaries of each adjacent valid original video frame interval are compatible with the latent variable time step boundaries.
[0123] When the above verification passes, multiple adjacent segmented pre-calculated latent variables can be concatenated along the time dimension of the latent variables according to the time order of each valid original video frame interval, and the concatenation result can be used as the segmented pre-calculated latent variable corresponding to the target original video frame interval. When the above verification fails or any segmented pre-calculated latent variable to be combined is unavailable, it can be determined that the first-level loading path has experienced a first recoverable failure.
[0124] In some optional implementations, when reading the segmented pre-calculated latent variables corresponding to the target original video frame interval through the first-level loading path, the following (B1)-(B3) may also be included.
[0125] (B1) The moment when the actual reading of the segmented pre-calculated latent variables begins is determined as the first reading start time.
[0126] Specifically, the first read time budget is calculated from the moment when the segmented pre-computed latent variables are actually read, rather than from the moment the current training data request is generated, the first-level availability index is queried, or other preprocessing is performed.
[0127] The actual start of a read operation can be any moment when a read call is initiated to the local file system, a read call to the network file system, or a data retrieval request is sent to object storage. The time spent on first-level availability index queries, configuration field preparation, and other processing before actual data reads are initiated can be excluded from the first read time budget.
[0128] (B2) The time elapsed since the start of the first reading time is defined as the first budget cutoff time.
[0129] Specifically, the first budget cutoff time can be obtained by adding the first read start time to the first read time budget.
[0130] In some alternative implementations, the first budget deadline can be controlled by a timer, an asynchronous task deadline, or a timeout parameter for a read request.
[0131] (B3) When the segmented pre-calculated latent variables are read no later than the first budget deadline, it is determined that the segmented pre-calculated latent variables have been successfully read within the first reading time budget; when the segmented pre-calculated latent variables have not been read by the first budget deadline, it is determined that the first reading timeout has occurred.
[0132] Specifically, if the complete segmented pre-computed latent variables are obtained before or just before the first budget deadline, the read can be considered successful. If the complete segmented pre-computed latent variables are not obtained by the first budget deadline, the corresponding read request can be canceled, subsequent return results ignored, or the read operation can be moved to the background for cleanup, and the second-layer loading path can be immediately entered. The first read timeout is used to limit the maximum time the current training step waits for the first-layer loading path, and does not require the corresponding storage object to be marked as permanently unavailable.
[0133] Step S203: When the first recovery failure or first read timeout occurs in the first loading path, switch to the second loading path, read the long interval pre-calculated latent variable covering the target original video frame interval through the second loading path, and when the long interval pre-calculated latent variable is successfully read within the second read time budget, perform latent variable time dimension slicing on the long interval pre-calculated latent variable to obtain the target latent variable.
[0134] Specifically, if the first recoverable failure or the first read timeout occurs in the first layer loading path, the current training step is not directly interrupted, but the second layer loading path can be tried.
[0135] The long-interval pre-computed latent variables stored in the second-layer loading path have a coverage range greater than or equal to the target original video frame interval. Therefore, if the reading is successful and the target boundary is compatible with the latent variable time grid, the target latent variables required for the current training step can be extracted from the long-interval pre-computed latent variables.
[0136] In some optional implementations, the long interval pre-computed latent variables covering the target original video frame interval are read through the second-layer loading path, and when the long interval pre-computed latent variables are successfully read within the second reading time budget, the latent variable time dimension slice is performed on the long interval pre-computed latent variables to obtain the target latent variables. This can be achieved in the following ways.
[0137] refer to Figure 3 , Figure 3 This is an exploded flowchart of an embodiment of step S203 according to this application. Step S203 may include the following steps S2031-S2036.
[0138] Step S2031: Obtain the second-level availability index.
[0139] The second-level availability index is used to characterize the correspondence between the generated long-interval pre-computed latent variables and the trajectory unit identifier, the effective original video frame interval, and the availability status of the long-interval pre-computed latent variables.
[0140] The second-level loading path can sequentially query the second-level availability index, read the long interval pre-calculated latent variables, determine the frame interval offset, determine the latent variable time index range, and perform latent variable time dimension slicing according to the latent variable time index range.
[0141] Specifically, the second-level availability index can use the same or different data structure as the first-level availability index.
[0142] In some alternative implementations, a second-level availability index record may include not only the trajectory unit identifier, valid raw video frame range, and availability status, but also the storage location, metadata location, and latent variable time grid information.
[0143] The second-level availability index is used to find long-interval pre-computed latent variables that cover the target's original video frame range.
[0144] Step S2032: Based on the trajectory unit identifier and the target original video frame interval, query the second-level availability index to determine the long interval pre-calculated latent variable that covers the target original video frame interval of the effective original video frame interval. The long interval pre-calculated latent variable is either the pre-calculated latent variable corresponding to the complete robot task trajectory data unit, or the pre-calculated latent variable corresponding to a longer continuous original video frame interval that covers the target original video frame interval and has a coverage range greater than the target original video frame interval.
[0145] Specifically, "the effective original video frame interval covering the target original video frame interval" means that the starting boundary of the effective original video frame interval is no later than the starting boundary of the target original video frame interval, and the ending boundary of the effective original video frame interval is no earlier than the ending boundary of the target original video frame interval. The long interval pre-calculated latent variables can cover either the complete robot task trajectory data unit or only one relatively long continuous interval, as long as the relatively long continuous interval completely covers the target original video frame interval of this request. This application does not require that the second-level loading path can only store the pre-calculated latent variables corresponding to the complete robot task trajectory data unit.
[0146] Step S2033: When the second-level availability index indicates that the corresponding long-interval pre-calculated latent variable is in an available state, the corresponding long-interval pre-calculated latent variable is read through the second-level loading path.
[0147] Specifically, the availability of long-range pre-computed latent variables indicates that the corresponding latent variables have been generated and stored, the second-level availability index has been updated, and the latent variable has not been isolated. The corresponding long-range pre-computed latent variables can be read based on their storage location in the second-level availability index, and the valid original video frame range and latent variable time grid for performing time-dimensional slicing can be obtained.
[0148] Step S2034: When the corresponding long interval pre-calculated latent variable is successfully read within the second reading time budget, determine the frame interval offset of the target original video frame interval relative to the starting point of the effective original video frame interval corresponding to the long interval pre-calculated latent variable.
[0149] Specifically, frame interval offset can include start boundary offset and end boundary offset. The start boundary offset represents the frame position difference between the start boundary of the target original video frame interval and the start point of the valid original video frame interval; the end boundary offset represents the frame position difference between the end boundary of the target original video frame interval and the start point of the valid original video frame interval.
[0150] For example, if the effective original video frame interval corresponding to the long interval pre-calculated latent variable is from frame 100 to frame 180, and the target original video frame interval is from frame 120 to frame 152, then the offset of the starting boundary relative to the starting point of the effective original video frame interval is 20 frames, and the offset of the ending boundary relative to the starting point of the effective original video frame interval is 52 frames.
[0151] Step S2035: Determine the latent variable time index range corresponding to the target original video frame interval based on the frame interval offset.
[0152] Specifically, the frame positions of the original video frame interval cannot be directly converted into latent variable time indices using a fixed division in all cases. Instead, a combination of pre-calculated latent variable time grids corresponding to long intervals can be used to determine whether the start and end boundaries of the target original video frame interval correspond to latent variable time step boundaries, and the range of latent variable time indices can be determined when the boundaries are compatible.
[0153] Step S2036: According to the time index range of the latent variables, the target latent variable is extracted from the pre-calculated latent variables over a long interval.
[0154] Specifically, the latent variables corresponding to the target original video frame interval can be sliced using the starting latent variable time index as the starting point and the ending latent variable time index as the ending point, based on the time dimension of the pre-computed latent variables over a long interval. When using the indexing rule that the starting index includes the latent variable time step but the ending index does not, the sliced latent variables include the latent variable time step corresponding to the starting latent variable time index, but do not include the latent variable time step corresponding to the ending latent variable time index.
[0155] In some optional implementations, the latent variable time index range corresponding to the target original video frame interval is determined based on the frame interval offset, which can be achieved in the following ways.
[0156] Specifically, the following method is used to convert the frame interval offset, represented by the starting point of the effective original video frame interval, into a latent variable time index range that can be directly used to extract long interval pre-computed latent variables.
[0157] First, the latent variable time grid corresponding to the long-interval pre-computed latent variables is obtained. The latent variable time grid is used to characterize the correspondence between each latent variable time step in the long-interval pre-computed latent variables and the original video frame range.
[0158] Specifically, the latent variable time grid can include multiple latent variable time step boundaries arranged in chronological order. Each latent variable time step boundary corresponds to an original video frame position relative to the start of the effective original video frame interval, and the range of original video frames between two adjacent latent variable time step boundaries corresponds to one latent variable time step in the long interval pre-computed latent variables. Therefore, the latent variable time grid is used to characterize the correspondence between each latent variable time step and the range of original video frames.
[0159] Latent variable time grids can be directly recorded from metadata of pre-computed latent variables over long intervals, or they can be determined based on the effective original video frame intervals, temporal downsampling factors, encoding start phase, and boundary processing methods. Using a latent variable time grid can further reflect the impact of the encoding start position or boundary processing method on temporal correspondences.
[0160] Then, based on the frame interval offset, the starting and ending boundaries of the target original video frame interval are determined to correspond to the latent variable time step boundaries in the latent variable time grid.
[0161] Specifically, the start boundary offset and end boundary offset can be matched with multiple latent variable time step boundaries in the latent variable time grid. If the start boundary offset matches one of the latent variable time step boundaries, then that latent variable time step boundary is the latent variable time step boundary corresponding to the start boundary. If the end boundary offset matches one of the latent variable time step boundaries, then that latent variable time step boundary is the latent variable time step boundary corresponding to the end boundary.
[0162] Next, based on the latent variable time step boundaries corresponding to the start and end boundaries, the start and end latent variable time indices are determined.
[0163] Specifically, the sequential position of the latent variable time step boundary in the latent variable time grid can be used to determine the corresponding latent variable time index.
[0164] For example, the starting boundary of the fifth latent variable time step can correspond to the starting latent variable time index 5, and the starting boundary of the thirteenth latent variable time step can correspond to the ending latent variable time index 13. Whether the actual index starts from 0 or 1 can be determined according to the indexing rules used, as long as the request side, metadata, and slicing process use the same indexing rules.
[0165] Finally, based on the starting and ending latent variable time indices, the range of latent variable time indices corresponding to the target original video frame interval is determined.
[0166] Specifically, when using the left-closed, right-open indexing rule, the index range that includes the starting latent variable time index but excludes the ending latent variable time index can be defined as the latent variable time index range.
[0167] For example, the frame offsets corresponding to the latent variable time step boundaries are 0, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, and 80, respectively. The starting boundary offset of 20 and the ending boundary offset of 52 corresponding to the target original video frame interval match the latent variable time step boundaries, so the starting latent variable time index is 5 and the ending latent variable time index is 13, and the latent variable time index range can be [5, 13).
[0168] If the start or end boundary of the target original video frame interval cannot match the latent variable time step boundary, then the target original video frame interval is incompatible with the latent variable time grid. In this case, the latent variable time index range that accurately corresponds to the target original video frame interval cannot be obtained by direct slicing. It can be determined that the second-level loading path has experienced a second recoverable failure, and the third-level loading path will continue to be attempted.
[0169] When reading long-interval pre-calculated latent variables covering the original video frame interval of the target through the second-level loading path, the following (C1)-(C3) may also be included: (C1) The moment when the actual reading of the long interval pre-calculated latent variables is determined as the second reading start time.
[0170] Specifically, the second read time budget is calculated from the moment when the long interval pre-computed latent variables are actually read, rather than from the moment when the first-level load path fails, the second-level availability index is queried, or other second-level preprocessing is performed.
[0171] The actual start of a read can be the moment a long-range pre-computed latent variable read call or data retrieval request is initiated to the corresponding storage backend. The time spent on second-level availability index queries and pre-processing before actual data reading is initiated can be excluded from the second read time budget.
[0172] (C2) The time elapsed since the second reading start time is defined as the second budget cutoff time.
[0173] Specifically, the second budget cutoff time can be obtained by adding the second read start time to the second read time budget. The first budget cutoff time and the second budget cutoff time correspond to the read processes of different loading paths and do not share the same timing start time or the same budget value.
[0174] (C3) When the long interval pre-calculated latent variable is read no later than the second budget deadline, it is determined that the long interval pre-calculated latent variable has been successfully read within the second reading time budget; when the long interval pre-calculated latent variable has not been read by the second budget deadline, it is determined that the second reading timeout has occurred.
[0175] Specifically, if the complete long-interval pre-computed latent variables are obtained no later than the second budget deadline, the latent variable time dimension slicing can continue. If the reading is not completed by the second budget deadline, the corresponding reading request can be canceled, subsequent return results ignored, or the reading operation can be moved to the background for cleanup and proceed to the third-layer loading path. The second reading timeout is used to limit the maximum time the current training step waits for the second-layer loading path.
[0176] Step S204: When the second recovery failure or second read timeout occurs in the second layer loading path, switch to the third layer loading path, read the original video frame sequence corresponding to the target original video frame interval, and use the video variational autoencoder currently used for training to perform online encoding on the original video frame sequence to obtain online encoding latent variables, and determine the online encoding latent variables as target latent variables.
[0177] Specifically, the third-layer loading path does not rely on existing pre-computed latent variables, but instead requests to read the original video frame sequence based on the current training data and performs online encoding using the video variational autoencoder used in the current training.
[0178] Therefore, when neither the first-level loading path nor the second-level loading path can return usable latent variables, the third-level loading path can serve as a guaranteed path for the supply of latent variables.
[0179] When the current training data request includes a set of target cameras, the original video frame sequences of each target camera within the target's original video frame range can be read separately, and online encoding can be performed using the video variational autoencoder version, temporal downsampling factor, and preprocessing version corresponding to the current training data request. Therefore, the latent variables of the online encoding can remain consistent with the current training configuration.
[0180] In one optional implementation, to reduce the impact of video variational autoencoder encoding at the boundary of the target original video frame interval, original video frames used to provide encoding context can be read before and after the target original video frame interval. The expanded original video frame sequence is then input into the video variational autoencoder, and after encoding, the latent variables corresponding to the context original video frames are removed, leaving only the online encoding latent variables corresponding to the target original video frame interval.
[0181] Online encoding itself may fail due to unreadable original video frame sequences, malfunctioning video variational autoencoders, or insufficient computational resources. Such failures are not classified as first-recoverable or second-recoverable failures. Retrying, skipping the current training sample, or terminating the training job can be performed according to training fault tolerance rules, and the specific reason for the error should be recorded.
[0182] In one optional implementation, online encoded latent variables can be written to a short-term cache, with at least one of the following as cache query conditions: trajectory unit identifier, target original video frame interval, and current training configuration. When subsequent training data requests match the cache query conditions, the online encoded latent variables in the short-term cache can be read directly. The short-term cache can be set with an expiration period or a capacity limit, and cache contents can be cleared according to a preset eviction policy. Writing online encoded latent variables to the short-term cache is not equivalent to synchronously generating and persisting complete segmented pre-computed latent variables or long-interval pre-computed latent variables.
[0183] Step S205: Provide the target latent variable to the current training step.
[0184] Specifically, the target latent variable can be a segmented pre-computed latent variable read from the first-layer loading path, a latent variable extracted from a long-interval pre-computed latent variable read from the second-layer loading path, or an online encoded latent variable generated by the third-layer loading path. The loading path that successfully obtains the target latent variable can be called the currently successful loading path. The target latent variable can be written to the memory buffer, device storage, training data queue, or other location accessible to the current training step.
[0185] In some alternative implementations, providing the target latent variable to the current training step may include the following (D1)-(D4).
[0186] (D1) Obtain the camera channel information corresponding to the target latent variables obtained through the currently successfully loaded path.
[0187] Specifically, camera channel information is used to characterize which camera corresponds to each camera channel in the target latent variable and the arrangement of each camera channel in the target latent variable. Camera channel information can be recorded in the metadata of segmented pre-computed latent variables or long-interval pre-computed latent variables, or it can be generated according to the input camera order when performing online encoding in the third-layer loading path.
[0188] (D2) Based on the camera channel information, determine the camera channel corresponding to each target camera in the target camera set in the target latent variables.
[0189] Specifically, the camera identifiers in the target camera set can be matched with the camera identifiers in the camera channel information to obtain the channel index of each target camera in the target latent variable. The target camera set can include all cameras contained in the robot task trajectory data unit, or it can include only some of the cameras.
[0190] (D3) Select the latent variables of the camera channels corresponding to each target camera from the target latent variables, and arrange the selected latent variables according to the target camera channel order.
[0191] Specifically, when the target latent variables obtained from the currently successfully loaded path contain more camera channels than are required for the current training step, only the camera channels corresponding to the target camera set can be selected without re-performing video variational autoencoder encoding. Subsequently, the selected camera latent variables are rearranged according to the target camera channel order.
[0192] For example, the target latent variables obtained from the currently successfully loaded path include the camera channels corresponding to the rear-view camera, front-view camera, and left-view camera in sequence. The target camera set includes the front-view camera and the left-view camera, and the target camera channels are in the order of left-view camera, front-view camera. A third camera channel and a second camera channel can be selected and arranged in the order of left-view camera, front-view camera.
[0193] (D4) The sorted latent variables are provided as target latent variables to the current training step.
[0194] Specifically, after completing the selection and arrangement of camera channels, the target latent variables before camera channel processing are replaced with the arranged latent variables, and then provided to the current training step. Thus, regardless of whether the target latent variables come from the first, second, or third layer loading path, the latent variables provided to the current training step can have the same target camera set and target camera channel order as the current training data request.
[0195] In some alternative implementations, the first recoverable failure includes at least one of the following: Specifically, the first recoverable failure is an anomaly or situation where the first layer loading path fails to obtain segmented pre-computed latent variables that can be used for the current training step, but still allows the system to continue trying the second layer loading path.
[0196] (1) There are no segmented pre-calculated latent variables corresponding to the trajectory unit identifier and the target original video frame interval, or the corresponding segmented pre-calculated latent variables are in an unavailable state.
[0197] Specifically, this situation may include situations where segmented pre-computed latent variables have not yet been generated, the first-level availability index has not yet been updated, the corresponding file has been deleted, the corresponding file has not yet been written, or the corresponding file has been isolated. Such situations do not indicate that the original video frame sequence is unavailable, and therefore can be downgraded to the second-level loading path.
[0198] (2) The video variational autoencoder version, temporal downsampling factor or preprocessing version corresponding to the pre-calculated latent variables of the corresponding segments are inconsistent with the video variational autoencoder version, temporal downsampling factor or preprocessing version corresponding to the current training data request.
[0199] Specifically, even if segmented pre-computed latent variables already exist, they cannot be used as target latent variables for this training if the video variational autoencoder, temporal downsampling relationship, or video preprocessing method used to generate them is inconsistent with the current training configuration. This file can be applied to other training configurations, therefore this situation can be considered the first recoverable failure under the current request, rather than deeming the file corrupted.
[0200] (3) When reading the corresponding segment pre-calculated latent variables, a temporary network access exception, object storage rate limit or other temporary access exception occurs.
[0201] Specifically, temporary network access anomalies can include brief connection interruptions, requests for resets, or momentary network unavailability; object storage rate limiting can include limiting the access frequency or concurrency of the object storage service; other temporary access anomalies can include brief unavailability of the network file system, excessive load on the storage backend, or read anomalies that can be recovered in subsequent accesses. When the above anomalies occur, waiting for the first-level loading path can be stopped and switched to the second-level loading path.
[0202] The second recoverable failure includes at least one of the following: Specifically, the second recoverable failure is an exception or situation where the second-level loading path fails to obtain long-interval pre-computed latent variables that can be used to perform latent variable time dimension slices, but still allows the system to continue to the third-level loading path to perform online coding, or the usage conditions are not met.
[0203] (1) There is no long interval pre-calculated latent variable that corresponds to the trajectory unit identifier and covers the target original video frame interval, or the corresponding long interval pre-calculated latent variable is in an unavailable state.
[0204] Specifically, this situation may include situations where long-range pre-computed latent variables have not yet been generated, the second-level availability index has not yet been updated, the generated long-range pre-computed latent variables do not fully cover the target original video frame range, the corresponding file has not yet been written, or the corresponding file has been isolated.
[0205] (2) The video variational autoencoder version, temporal downsampling factor or preprocessing version corresponding to the pre-calculated latent variables in the corresponding long interval is inconsistent with the video variational autoencoder version, temporal downsampling factor or preprocessing version corresponding to the current training data request.
[0206] Specifically, this situation indicates that although the long-interval pre-computed latent variables exist and can cover the target original video frame interval in terms of frame range, their encoding semantics or preprocessing methods are inconsistent with the current training configuration, so the target latent variables required for this training cannot be obtained by slicing.
[0207] In one optional implementation, when the configuration of the long-interval pre-computed latent variables is inconsistent with the current training configuration, it can be first determined whether there is a pre-validated compatible transformation method for the configuration difference. Only when the compatible transformation method can maintain the semantics of the latent variables and make the transformed configuration consistent with the current training configuration, is the compatible transformation performed on the long-interval pre-computed latent variables or the latent variable portion to be sliced, and the latent variable time dimension slicing continues; if there is no pre-validated compatible transformation method or the transformation fails, it is still determined that the second-layer loading path has experienced a second recoverable failure and the process switches to the third-layer loading path.
[0208] (3) It is impossible to determine the latent variable time index range that can perform latent variable time dimension slicing on the corresponding long interval based on the target original video frame interval.
[0209] Specifically, this situation may include the starting or ending boundary of the target original video frame interval not matching the latent variable time step boundary in the latent variable time grid, missing latent variable time grid information, or an incomplete correspondence between the valid original video frame interval and the latent variable time dimension length. In this case, direct slicing may yield latent variables that do not accurately correspond to the target original video frame interval, thus the system enters the third-level loading path.
[0210] (4) When reading the corresponding long interval pre-calculated latent variables, a temporary network access exception, object storage rate limit or other temporary access exception occurs.
[0211] Specifically, this situation can be understood by referring to the temporary access exception in the first-layer loading path. Since the third-layer loading path can perform online encoding based on the original video frame sequence, the second-layer loading path can continue to degrade when a temporary access exception occurs, without having to immediately interrupt the current training step.
[0212] It should be noted that the first read timeout and the second read timeout are treated as degradation conditions alongside the first recoverable failure and the second recoverable failure, respectively. The read timeout can be determined by the aforementioned first read time budget and second read time budget. The soft degradation in this application refers to continuing to try the next layer of loading path when a recoverable failure or read timeout occurs in a high-priority loading path, rather than immediately interrupting training.
[0213] In some alternative implementations, the following (E1)-(E7) may also be included before determining the segmented pre-computed latent variables obtained through the first-level loading path as the target latent variables, or before performing latent variable time dimension slicing on the long-interval pre-computed latent variables obtained through the second-level loading path: Specifically, both the first and second layer loading paths obtain pre-generated latent variables. To avoid using latent variables generated by older versions of the video variational autoencoder, latent variables corresponding to incorrect camera configurations, or latent variables corresponding to incorrect time ranges in the current training step, configuration consistency checks, target range coverage checks, and data integrity checks can be performed before using the pre-computed latent variables.
[0214] (E1) Determine the request consistency field corresponding to the current training data request. The request consistency field includes at least one of the following configuration fields: trajectory unit identifier, target range, camera configuration information, encoding configuration information, video preprocessing information, and latent variable format information. The target range includes the target original video frame range or its corresponding target latent variable time range.
[0215] Specifically, the trajectory unit identifier is used to determine whether the latent variables originate from the robot task trajectory data unit corresponding to this request. Camera configuration information may include the target camera set and the target camera channel order. Encoding configuration information may include the video variational autoencoder identifier and version, and the temporal downsampling factor. Video preprocessing information may include video resolution, image normalization method, image cropping method, or preprocessing version. Latent variable format information may include the data type, dimensions, tensor axis order, or boundary handling method of the latent variables.
[0216] The trajectory unit identifier and target range in the aforementioned consistency field are used to determine the training data object and its target interval; camera configuration information, encoding configuration information, video preprocessing information, and latent variable format information can be selected from at least one of these configuration fields according to specific verification requirements. In actual implementation, more fields can be used for verification, but it is not required that all listed fields be used simultaneously in every verification.
[0217] (E2) Extract the segmented consistency field corresponding to the requested consistency field from the metadata of the segmented pre-computed latent variables, or determine the long interval consistency field corresponding to the part of the latent variable to be cut out based on the metadata of the long interval pre-computed latent variables.
[0218] Specifically, when the object to be verified is a segmented pre-calculated latent variable, the trajectory unit identifier, valid original video frame interval, camera information, encoding configuration, video preprocessing information, and latent variable format information can be read from the metadata of the segmented pre-calculated latent variable to form segmented consistency fields that correspond to each item of the requested consistency fields. When the object to be verified is a long interval pre-calculated latent variable, the target range of the part to be cut cannot be replaced by the range field of the entire long interval. Instead, the long interval consistency field corresponding to the part of the latent variable to be cut can be determined based on the metadata of the long interval pre-calculated latent variable, the frame interval offset, and the latent variable time index range to be used.
[0219] (E3) Generate a request configuration fingerprint based on the request consistency field according to the same field order and deterministic serialization rules, and generate a segmented configuration fingerprint or a long interval configuration fingerprint based on the segmented consistency field or the long interval consistency field.
[0220] Specifically, configuring a fingerprint can be expressed as formula (2): (2) Among them, Metadata represents request consistency fields, segmented consistency fields, or long interval consistency fields organized according to a preset field order; Serialize represents converting the corresponding fields into a byte sequence using deterministic serialization rules; Hash represents hash operation; and Fingerprint represents the generated configuration fingerprint.
[0221] Deterministic serialization rules can specify field order, character encoding, default value representation, numeric format, array element order, and interval boundary representation. Requested consistency fields and their corresponding pre-computed latent variable consistency fields use the same field order and deterministic serialization rules, ensuring that fields with the same semantic configuration can generate the same configuration fingerprint.
[0222] (E4) Compare the requested configuration fingerprint with the segmented configuration fingerprint or the long interval configuration fingerprint, and perform target range coverage verification and data integrity verification on the segmented pre-computed latent variables or the long interval pre-computed latent variables.
[0223] Specifically, when the object to be verified is a segmented pre-computed latent variable, the request configuration fingerprint is compared with the segmented configuration fingerprint. When the object to be verified is a long-interval pre-computed latent variable, the request configuration fingerprint is compared with the long-interval configuration fingerprint. Configuration fingerprint comparison is used to quickly determine whether the semantic configuration between the training request and the pre-computed latent variable is consistent, but it cannot replace target range coverage verification and data integrity verification.
[0224] Target coverage verification is used to determine whether segmented pre-calculated latent variables or long-interval pre-calculated latent variables completely cover the target's original video frame interval or the target latent variable's time range. Data integrity verification can include data content verification and validation, object version identifier verification, deserialization result verification, latent variable dimension verification, trajectory unit identifier conflict verification, or security verification.
[0225] (E5) When the configuration fingerprint is consistent, covers the target range and passes the data integrity check, determine that the segmented pre-calculated latent variables meet the conditions for use as target latent variables, or determine that the long interval pre-calculated latent variables meet the conditions for use in executing latent variable time dimension slicing.
[0226] Specifically, when a segmented pre-computed latent variable meets the above conditions, it can be identified as a target latent variable after it is successfully read within the first read time budget. When a long-interval pre-computed latent variable meets the above conditions, it means that the long-interval pre-computed latent variable is suitable for this training request in terms of semantic configuration, coverage, and data content, and time dimension slicing can continue to be performed based on the time index range of the latent variable.
[0227] (E6) When the configured fingerprint is inconsistent or does not cover the target range, the first recoverable failure occurs when determining the first layer loading path for segmented pre-calculated latent variables, or the second recoverable failure occurs when determining the second layer loading path for long interval pre-calculated latent variables.
[0228] Specifically, an inconsistent configuration fingerprint indicates that the corresponding pre-computed latent variable is not applicable to the current training configuration, but it may still be applicable to other training configurations. An uncovered target range indicates that the pre-computed latent variable cannot provide the complete time range required for this request. These situations do not necessarily indicate that the corresponding file is corrupted; therefore, they can be considered recoverable failures of the corresponding loading path, and the next loading path can be tried.
[0229] (E7) When the segmented pre-calculated latent variable fails the data integrity check, record and isolate the data integrity anomaly of the segmented pre-calculated latent variable, and switch to the second-level loading path; when the long interval pre-calculated latent variable fails the data integrity check, record and isolate the data integrity anomaly of the long interval pre-calculated latent variable, and switch to the third-level loading path.
[0230] Specifically, data integrity checks that fail may include inconsistencies in checksums, corrupted deserialization results, discrepancies between latent variable dimensions and metadata, conflicting trajectory unit identifiers, or security check failures. These are considered unrecoverable or alert-required anomalies. After recording the anomaly object, anomaly type, occurrence time, and request information, the anomalous data can be marked as isolated to prevent subsequent requests from using it. Even after recording and isolation, it's still possible to switch to the next loading path to maintain the current training process; however, data integrity anomalies cannot be ignored without being recorded.
[0231] After providing the target latent variable to the current training step, it is also possible to record the actual loading path used for this latent variable dynamic supply, the total loading delay, the reading delay of the first layer loading path and the second layer loading path, the reason why each loading path failed to return, the corresponding storage backend, and the trajectory unit identifier.
[0232] Historical delay estimates can be maintained according to at least one statistical dimension from the robot task trajectory data unit, loading path, or storage backend. Historical delay estimates can be updated using an exponential moving average method and can be expressed as formula (3): (3) in, Indicates the corresponding statistical object; This indicates the actual readout delay for this observation; This indicates the historical delay estimate before the update; ρ represents the updated historical delay estimate; ρ represents the update weight of the current observation, and satisfies 0 < ρ ≤ 1.
[0233] For example, if the historical latency estimate before the update was 100 milliseconds, the actual read latency this time was 160 milliseconds, and the update weight was 0.2, then the historical latency estimate after the update would be 112 milliseconds. Using weighted updates can avoid excessive changes in the historical latency estimate caused by a single sporadic network fluctuation.
[0234] The read latency of successfully reading segmented pre-computed latent variables can be added to the first latency sample set, and the read latency of successfully reading long-interval pre-computed latent variables can be added to the second latency sample set. This allows for dynamic updates to the first and second read time budgets based on recent successful read latency. Robot task trajectory data unit-level statistics can be used to determine which robot task trajectory data units frequently enter high-latency paths, while storage backend-level statistics can be used to determine whether the overall read performance of the corresponding storage backend has changed.
[0235] refer to Figure 4 , Figure 4 This is a flowchart of an embodiment of performing asynchronous preheating processing according to this application.
[0236] In some alternative implementations, after the online encoded latent variable is identified as the target latent variable, the process 400 may include the following steps S401-S403.
[0237] Specifically, online encoding of latent variables can be directly used in the current training step. To ensure that the same robot task trajectory data unit can use the lower-latency first-layer loading path or second-layer loading path when accessed again later, the usage of the third-layer loading path can be statistically analyzed, and a background pre-computation task can be submitted asynchronously when the conditions are met.
[0238] Step S401: Within the preset statistics window or since the start of training, accumulate the number of times the robot task trajectory data unit with trajectory unit identification enters the third-layer loading path for online encoding.
[0239] Specifically, the preset statistics window can be the most recent preset number of training steps, the most recent preset duration, or the most recent preset number of access records. Alternatively, it can continuously accumulate the number of online encoding attempts since the start of training. The object for accumulating online encoding attempts is the robot task trajectory data unit with the corresponding trajectory unit identifier. When the same robot task trajectory data unit enters the third-layer loading path multiple times in different training steps, its online encoding attempts increase accordingly.
[0240] The threshold condition for the number of online encoding attempts can be expressed as formula (4): (4) in, Indicates the robot task trajectory data unit number; Indicates the first The number of times each robot task trajectory data unit has entered the third-layer loading path online within a preset statistical window or since the start of training; This indicates the preset threshold for the number of online encoding attempts.
[0241] Step S402: Only after the number of online encoding attempts reaches the preset online encoding attempt threshold is the asynchronous preheating submission judgment performed.
[0242] Specifically, reaching the preset online coding count threshold only triggers the asynchronous pre-warm-up submission judgment and does not necessarily mean that the background pre-calculation task must be submitted. If the online coding count does not reach the threshold, the asynchronous pre-warm-up judgment can be terminated directly, thereby avoiding premature generation of pre-calculated latent variables for robot task trajectory data units that occasionally enter the third-level loading path.
[0243] Step S403: When the result of the asynchronous preheating submission judgment is that the preheating task submission conditions are met, after providing the target latent variable to the current training step, the background pre-computation task for generating the corresponding segmented pre-computation latent variable or long interval pre-computation latent variable is asynchronously submitted.
[0244] Specifically, the background pre-computation task can be used to generate segmented pre-computation latent variables corresponding to a target original video frame interval, or it can be used to generate long-interval pre-computation latent variables corresponding to a complete robot task trajectory data unit or a longer continuous original video frame interval. Asynchronous submission occurs after the target latent variables have been provided to the current training step, and the current training step does not wait for the background pre-computation task to complete. The execution result of the background pre-computation task only affects the loading path that subsequent training data requests for the corresponding robot task trajectory data unit can use.
[0245] In some alternative implementations, the asynchronous preheating submission decision includes the following (F1)-(F3).
[0246] Specifically, after the number of online coding attempts reaches a threshold, the expected latency benefits generated by performing pre-computation can be further compared with the equivalent cost required to perform pre-computation to determine whether it is worthwhile to submit a background pre-computation task.
[0247] (F1) Based on the sampling frequency of the corresponding robot task trajectory data unit and the expected number of remaining training steps, determine the expected number of visits to the robot task trajectory data unit in the remaining training process.
[0248] Specifically, the sampling frequency represents the frequency with which a corresponding robot task trajectory data unit is accessed within a unit of training steps. The estimated number of remaining training steps can be determined based on the current training progress, the preset total number of training steps, or the training termination condition. The estimated number of accesses can be obtained by multiplying the sampling frequency by the estimated number of remaining training steps.
[0249] (F2) Determine the expected net benefit of performing precomputation based on the expected number of accesses, the single-pass latency savings between online encoding and precomputation of latent variables, and the equivalent cost required to perform precomputation.
[0250] Specifically, single-instance latency savings represent the difference between the latency required for the same training data request to be encoded online and the latency required to read pre-computed latent variables. Pre-computed equivalent costs can include computational resources used for video variational autoencoder encoding, storage resources occupied by pre-computed latent variables, background task scheduling overhead or fees, and can be converted into a uniform unit comparable to latency gains.
[0251] The projected net income from the pre-calculation can be expressed as formula (5): (5) in, Indicates the first The sampling frequency of each robot task trajectory data unit; Indicates the expected number of remaining training steps; Indicates the expected number of visits; This represents the single-pass latency savings between online encoding and pre-computed latent variable reads; This represents the equivalent cost required to perform the pre-calculation; This indicates the expected net income.
[0252] For example, if the sampling frequency is 0.002 times per training step, the expected number of remaining training steps is 50,000, the single latency saving is 0.18 seconds, the pre-calculated equivalent cost is 8 seconds, then the expected number of accesses is 100, the expected latency benefit is 18 seconds, and the expected net benefit is 10 seconds.
[0253] (F3) If the expected net income meets the preset income conditions, the preheating task submission conditions are determined to be met; if the expected net income does not meet the preset income conditions, the preheating task submission conditions are determined not to be met.
[0254] Specifically, the preset revenue condition can be either an expected net revenue greater than 0 or an expected net revenue greater than a preset revenue threshold. When the expected net revenue meets the preset revenue condition, it means that the pre-calculated delayed revenue generated during the expected remaining training process is sufficient to compensate for its equivalent cost; when the expected net revenue does not meet the preset revenue condition, the corresponding background pre-calculation task does not need to be submitted.
[0255] After the projected net revenue meets the preset revenue conditions, the system can also determine whether the background resources allow the submission of a background pre-computation task based on the background computing resources, storage resources, or task queue load, and whether an equivalent background pre-computation task already exists. The corresponding background pre-computation task will only be submitted if the background resources allow it and no equivalent background pre-computation task exists.
[0256] When there are multiple submitted background pre-computation tasks, steps S403 may be followed by steps S404-S405.
[0257] Specifically, multiple background pre-computation tasks can be triggered by different robot task trajectory data units, or they can be used to generate different types of pre-computation results. Here, the execution order of multiple background pre-computation tasks can be determined when background computing resources are limited.
[0258] S404. Based on the sampling frequency, expected delay savings per attempt, probability of successful pre-calculation, and equivalent cost of pre-calculation of the robot task trajectory data unit corresponding to each background pre-calculation task, determine the scheduling priority of each background pre-calculation task.
[0259] Specifically, a higher sampling frequency indicates a higher probability or frequency of access to the corresponding pre-computation results during subsequent training; a greater expected latency saving per instance indicates a greater reduction in latency each time the online encoding is switched to pre-computation latent variable reading is performed; a higher probability of successful pre-computation indicates a higher likelihood that the background pre-computation task will successfully generate usable latent variables; and a higher equivalent cost of pre-computation indicates a greater resource requirement for executing the background pre-computation task.
[0260] Scheduling priority can be expressed as formula (6): (6) in, This indicates the number of the background pre-calculation task. Indicates the first The scheduling priority of each background pre-computation task; Indicates the first The recent sampling frequency of robot task trajectory data units corresponding to each background pre-calculation task; Indicates execution of the first The expected latency savings per task that can be generated after a background pre-computing task; Indicates the first One background pre-computation task successfully generated the estimated probability of usable pre-computation latent variables; Indicates the first The pre-computation equivalent cost of each background pre-computation task; This indicates a positive number used to avoid a denominator of zero.
[0261] When the costs and delays of different tasks are measured in a uniform dimension or after normalization, a higher scheduling priority indicates that the expected training delay benefit per unit cost of the task is higher.
[0262] S405 determines the execution order of multiple background pre-computation tasks according to their scheduling priority from high to low.
[0263] Specifically, the background scheduler can prioritize the execution of background pre-computation tasks with higher scheduling priority, and reorder the execution of remaining tasks after task completion, failure, or priority update. This execution order is used for resource allocation among background pre-computation tasks and does not change the execution state of the current training step that has already obtained online encoded latent variables.
[0264] In an alternative implementation, a task key can also be generated for the background pre-computation task. The task key can be generated based on the trajectory unit identifier, the pre-computation result type, the original video frame interval to be pre-computed, the video variational autoencoder version, the temporal downsampling factor, the target camera set, and the preprocessing version. The pre-computation result type indicates whether segmented pre-computation latent variables or long-interval pre-computation latent variables are generated.
[0265] Before submitting a background pre-computation task, you can query the task status of existing background pre-computation tasks based on the task key. The task status can include pending, processing, and completed. If an existing background pre-computation task with the same task key exists and is in the pending, processing, or completed state, you do not need to submit a new background pre-computation task; if no existing background pre-computation task with the same task key exists, you can submit a new background pre-computation task.
[0266] For example, two data loading processes might trigger online encoding of the same robot trajectory data unit almost simultaneously and prepare to submit background pre-computation tasks with the same configuration. After the first data loading process creates the corresponding task key and registers the task as pending, the second data loading process can find that the same task key already exists and therefore does not need to submit it again. This reduces redundant video variational autoencoder encoding and redundant storage writes.
[0267] In an alternative implementation, the processing order of corresponding data units can be adjusted within a short time window based on the historical delay estimate of the robot task trajectory data units. The temporary sampling weight can be expressed as formula (7): (7) in, Indicates the original sampling weights; This indicates the adjusted temporary sampling weights; Indicates the minimum temporary sampling weight that is allowed to be used; Indicates target loading delay; Indicates historical delay estimation; This indicates a positive number used to avoid a denominator of zero.
[0268] When the historical delay estimate is no greater than the target loading delay, the delay term does not increase the original sampling weight; when the historical delay estimate is greater than the target loading delay, the scheduling priority of the corresponding robot task trajectory data unit can be reduced within a short time window. This method is preferably used to adjust the processing order within the same training round or to postpone high-latency data to a period when the prefetch queue is sufficient, without permanently changing the distribution of training data in the absence of distribution correction.
[0269] Furthermore, when adjusting the processing order using delay-aware methods, the total number of samples for each robot task trajectory data unit within the same training round can remain unchanged. If the sampling probability of some robot task trajectory data units is reduced within a short time window, the corresponding training loss can be corrected using importance weights, and a minimum sampling probability can be set for each robot task trajectory data unit. After the corresponding pre-calculated latent variables are generated or the short time window ends, the original sampling distribution can be restored.
[0270] At the beginning of each training round, the robot task trajectory data units can be divided into ready data units and not ready data units based on whether there are valid pre-calculated latent variables in the first layer loading path or the second layer loading path. Ready data units are placed in the earlier prefetch window first, while a preset processing quota is reserved for not ready data units.
[0271] When the training sampling order is predictable, asynchronous prefetching can be initiated in advance for existing segmented pre-computed latent variables or long-interval pre-computed latent variables. This predictive prefetching targets existing pre-computed latent variables, aiming to overlap the data reading process with the model calculation process; asynchronous pre-warming targets pre-computed latent variables that have not yet been generated, aiming to trigger persistent pre-computation. These two are different processing procedures.
[0272] In some alternative implementations, the prefetch depth of predictive prefetching can be dynamically determined based on available memory or GPU memory, historical cache hit rate, and prediction confidence for subsequent training data requests. The more available memory or GPU memory, the higher the historical cache hit rate, or the higher the prediction confidence, the greater the prefetch depth can be. Conversely, when available memory or GPU memory is insufficient, the historical cache hit rate is low, or the prediction confidence is low, the prefetch depth can be reduced or predictive prefetching can be paused to limit invalid reads and cache occupancy.
[0273] In one alternative implementation, the pre-calculated coverage within a future training window can also be predicted based on the historical sampling frequency of the robot task trajectory data unit.
[0274] The probability that the i-th robot task trajectory data unit, which has not yet completed pre-calculation, will be sampled at least once within the prediction window can be expressed as formula (8): (8) in, Indicates the first Sampling frequency of a unit training step for a robot task trajectory data unit; This indicates the number of training steps contained in the prediction window; Indicates the first The probability that a robot mission trajectory data unit is sampled at least once within the prediction window.
[0275] The expected number of new data units entering the pre-computation ready state within the prediction window can be expressed as formula (9): (9) in, Indicates that in the inclusion The number of robot task trajectory data units added to the pre-computation ready state within the prediction window of each training step. This represents the expected value of the newly added quantity; This represents the collection of all robot task trajectory data units; This represents a set of robot task trajectory data units that already have valid pre-computed latent variables; This represents a set of robot task trajectory data units that do not yet have valid pre-computed latent variables.
[0276] The predicted pre-calculated coverage can be expressed as formula (10): (10) in, Indicates that it has been included Pre-computed coverage of predictions after the prediction window of each training step; This indicates the number of robot task trajectory data units that currently have valid pre-computed latent variables; This represents the expected number of new data units entering the pre-computation ready state within the prediction window; This represents the total number of robot task trajectory data units.
[0277] When the sampling process does not meet the conditions of approximate independence and stationarity, the sliding window empirical probability or exponential decay counting can be used to replace the above probability models. When the prediction pre-computation coverage grows slowly and the proportion of online encoding is still high, the backend pre-computation concurrency can be increased or the online encoding threshold can be reduced; when backend resources are scarce and the expected returns are low, the backend pre-computation concurrency can be reduced.
[0278] In an alternative implementation, the expected marginal benefit, i.e. the expected net benefit, for robot task trajectory data units that have not yet been pre-calculated can be expressed as formula (11): (11) in, This indicates the expected number of visits to the corresponding robot task trajectory data unit during the remaining training process; when determining the expected number of visits based on the sampling frequency and the expected number of remaining training steps, Therefore, formula (11) is an updated calculation of the expected net profit in formula (5) based on the current training state.
[0279] When the expected marginal returns of candidate tasks calculated in the background do not meet the preset conditions, the background pre-calculation can be paused or stopped. Therefore, the background pre-calculation does not need to use the completion of pre-calculation for all robot task trajectory data units as a fixed stopping condition, but can dynamically determine the stopping condition based on the actual sampling distribution, the expected number of remaining training steps, and resource costs.
[0280] In an alternative implementation, the expected read latency or success probability of the first and second layer loading paths for the current training data request can be predicted based on the historical hit rate, historical read latency, corresponding storage backend status, and current network status of the first and second layer loading paths. Based on this, it can be determined whether to initiate parallel probing for the lower priority pre-computed loading path in advance, or to adjust the attempt order between the first and second layer loading paths.
[0281] The above predictions can be achieved using pre-defined rules, statistical models, or learning models, and the use of a reinforcement learning model is not a necessary condition. Regardless of whether the above predictions are used, the third-layer loading path serves as the final backup path; when the pre-computed loading path fails to obtain the target latent variable, online encoding is still performed through the third-layer loading path to ensure that the supply of training data can continue.
[0282] This application attempts to acquire latent variables sequentially using three loading paths: a first-level loading path, a second-level loading path, and a third-level loading path. When a high-priority loading path fails to recover or times out, it switches to the next loading path. This allows the visual-language-action model training to continue acquiring target latent variables even when the pre-computed latent variables haven't fully covered the training data. The first-level loading path directly reads segmented pre-computed latent variables corresponding to the target original video frame interval. The second-level loading path extracts the target latent variable from long-interval pre-computed latent variables covering the target original video frame interval. The third-level loading path performs online encoding on the original video frame sequence when the first two paths fail. Therefore, while prioritizing the use of low-latency pre-computed results, online encoding ensures data availability for the current training step. By setting separate first and second read time budgets, the current training step can avoid continuously waiting for pre-computed latent variables with excessively slow read speeds. By configuring consistency checks, target range coverage checks, and data integrity checks, the risk of latent variables returning semantically inconsistent or corrupted data from different loading paths can be reduced. By using online encoding count statistics, expected net profit judgment, and asynchronous submission of background pre-computed tasks, subsequent training data requests can gradually use pre-computed latent variables with lower latency without blocking the current training step.
[0283] refer to Figure 5 , Figure 5 A schematic diagram of one embodiment of the latent variable dynamic supply device according to this application is shown.
[0284] like Figure 5 As shown, the latent variable dynamic supply device 500 includes a request acquisition module 501, a first-layer loading module 502, a second-layer loading module 503, a third-layer loading module 504, and a latent variable supply module 505.
[0285] The request acquisition module 501 is used to acquire the current training data request, which includes the trajectory unit identifier and the target original video frame range.
[0286] The first-layer loading module 502 is used to read the segmented pre-calculated latent variables corresponding to the target original video frame interval through the first-layer loading path according to the trajectory unit identifier and the target original video frame interval, and to determine the segmented pre-calculated latent variables as target latent variables when the segmented pre-calculated latent variables are successfully read within the first reading time budget.
[0287] The second-layer loading module 503 is used to read long-range pre-calculated latent variables covering the target original video frame range through the second-layer loading path when the first recoverable failure or the first reading timeout occurs in the first-layer loading path, and to perform latent variable time dimension slicing on the long-range pre-calculated latent variables to obtain the target latent variables when the long-range pre-calculated latent variables are successfully read within the second reading time budget.
[0288] The third-layer loading module 504 is used to read the original video frame sequence corresponding to the target original video frame interval when the second recoverable failure or the second reading timeout occurs in the second-layer loading path, and to perform online encoding on the original video frame sequence using the currently trained video variational autoencoder to obtain online encoding latent variables, and to determine the online encoding latent variables as target latent variables.
[0289] The latent variable providing module 505 is used to provide the target latent variable to the current training step.
[0290] In some alternative implementations, the first-layer loading module 502 is further configured to: Obtain the first-level availability index, which is used to characterize the correspondence between the generated segmented pre-computed latent variables and the trajectory unit identifier, the effective original video frame interval, and the availability status of the segmented pre-computed latent variables; Based on the trajectory unit identifier and the target original video frame interval, query the first-level availability index to determine the segmented pre-calculated latent variables whose start and end boundaries of the valid original video frame interval are the same as the start and end boundaries of the target original video frame interval; When the first-level availability index indicates that the corresponding segment pre-computed latent variable is available, the corresponding segment pre-computed latent variable is read through the first-level loading path.
[0291] In some alternative implementations, the second-layer loading module 503 is further configured to: Obtain the second-level availability index, which is used to characterize the correspondence between the generated long-interval pre-computed latent variables and the trajectory unit identifier, the effective original video frame interval, and the availability status of the long-interval pre-computed latent variables; Based on the trajectory unit identifier and the target original video frame interval, query the second-level availability index to determine the long interval pre-calculated latent variable that the effective original video frame interval covers the target original video frame interval. The long interval pre-calculated latent variable is the pre-calculated latent variable corresponding to the complete robot task trajectory data unit, or it is the pre-calculated latent variable corresponding to a longer continuous original video frame interval that covers the target original video frame interval and has a coverage range greater than the target original video frame interval. When the second-level availability index indicates that the corresponding long-interval pre-computed latent variable is available, the corresponding long-interval pre-computed latent variable is read through the second-level loading path. When the corresponding long interval pre-calculated latent variable is successfully read within the second reading time budget, determine the frame interval offset of the target original video frame interval relative to the starting point of the effective original video frame interval corresponding to the long interval pre-calculated latent variable; Based on the frame interval offset, determine the range of latent variable time indexes corresponding to the target original video frame interval; Based on the time index range of the latent variables, the target latent variable is extracted from the pre-calculated latent variables over a long interval.
[0292] In some optional implementations, the second-layer loading module 503, when determining the latent variable time index range corresponding to the target original video frame interval based on the frame interval offset, is further configured to: Obtain the latent variable time grid corresponding to the long interval pre-computed latent variable. The latent variable time grid is used to characterize the correspondence between each latent variable time step in the long interval pre-computed latent variable and the original video frame range. Based on the frame interval offset, determine the latent variable time step boundaries corresponding to the start and end boundaries of the target original video frame interval in the latent variable time grid. The start latent variable time index and the end latent variable time index are determined based on the latent variable time step boundaries corresponding to the start boundary and the end boundary, respectively. Based on the starting and ending latent variable time indices, determine the range of latent variable time indices corresponding to the target original video frame interval.
[0293] In some alternative implementations, the current training data request may also include the target camera set and the target camera channel order.
[0294] The latent variable providing module 505, when providing the target latent variable to the current training step, is further used for: Obtain camera channel information corresponding to the target latent variables obtained through the currently successfully loaded path; Based on the camera channel information, determine the camera channel corresponding to each target camera in the target camera set in the target latent variables; Select the latent variables of the camera channels corresponding to each target camera from the target latent variables, and arrange the selected latent variables according to the target camera channel order; The sorted latent variables are used as target latent variables and provided to the current training step.
[0295] In some optional implementations, the current training data request may also include encoding configuration information, which includes the video variational autoencoder version, the temporal downsampling factor, and the preprocessing version.
[0296] The first recoverable failure includes at least one of the following: There are no segmented pre-calculated latent variables corresponding to the trajectory unit identifier and the target original video frame interval, or the corresponding segmented pre-calculated latent variables are in an unavailable state; The video variational autoencoder version, temporal downsampling factor, or preprocessing version corresponding to the pre-computed latent variables in the corresponding segments are inconsistent with the video variational autoencoder version, temporal downsampling factor, or preprocessing version corresponding to the current training data request. Temporary network access exceptions, object storage rate limiting, or other temporary access exceptions occurred when reading the corresponding segmented pre-calculated latent variables.
[0297] The second recoverable failure includes at least one of the following: There are no long-interval pre-calculated latent variables that correspond to the trajectory unit identifier and whose valid original video frame intervals cover the target original video frame intervals, or the corresponding long-interval pre-calculated latent variables are in an unavailable state; The video variational autoencoder version, temporal downsampling factor, or preprocessing version corresponding to the pre-computed latent variables over the corresponding long interval is inconsistent with the video variational autoencoder version, temporal downsampling factor, or preprocessing version corresponding to the current training data request. It is impossible to determine the range of latent variable time indexes that can be used to perform latent variable time dimension slicing on the corresponding long interval based on the target original video frame interval; Temporary network access exceptions, object storage rate limiting, or other temporary access exceptions may occur when reading the corresponding long interval pre-calculated latent variables.
[0298] In some optional implementations, before determining the segmented pre-computed latent variables obtained through the first-level loading path as the target latent variables, or before performing latent variable time dimension slicing on the long-interval pre-computed latent variables obtained through the second-level loading path, the first-level loading module 502 or the second-level loading module 503 is further configured to: Determine the request consistency field corresponding to the current training data request. The request consistency field includes at least one of the following configuration fields: trajectory unit identifier, target range, camera configuration information, encoding configuration information, video preprocessing information, and latent variable format information. The target range includes the target original video frame range or its corresponding target latent variable time range. Extract the segmented consistency field corresponding to the requested consistency field from the metadata of the segmented pre-computed latent variables, or determine the long interval consistency field corresponding to the part of the latent variable to be cut out based on the metadata of the long interval pre-computed latent variables. Following the same field order and deterministic serialization rules, a request configuration fingerprint is generated based on the request consistency field, and a segmented configuration fingerprint or a long interval configuration fingerprint is generated based on the segmented consistency field or the long interval consistency field. Compare the requested configuration fingerprint with the segmented configuration fingerprint or the long interval configuration fingerprint, and perform target range coverage verification and data integrity verification on the segmented pre-computed latent variables or the long interval pre-computed latent variables; When the configuration fingerprint is consistent, covers the target range, and passes the data integrity verification, determine that the segmented pre-calculated latent variables meet the conditions for use as target latent variables, or determine that the long interval pre-calculated latent variables meet the conditions for use in executing latent variable time dimension slicing. When the configured fingerprint is inconsistent or does not cover the target range, the first recoverable failure occurs when determining the first layer loading path for segmented pre-calculated latent variables, or the second recoverable failure occurs when determining the second layer loading path for long interval pre-calculated latent variables. When the segmented pre-calculated latent variable fails the data integrity check, record and isolate the data integrity anomaly of the segmented pre-calculated latent variable, and switch to the second-level loading path; when the long-interval pre-calculated latent variable fails the data integrity check, record and isolate the data integrity anomaly of the long-interval pre-calculated latent variable, and switch to the third-level loading path.
[0299] In some optional implementations, when the first-layer loading module 502 reads the segmented pre-calculated latent variables corresponding to the target original video frame interval through the first-layer loading path, it is further used for: The moment when the actual reading of segmented pre-calculated latent variables begins is defined as the first reading start moment; The time elapsed since the start of the first reading and the first reading time budget is defined as the first budget cutoff time. If the segmented pre-calculated latent variables are read no later than the first budget deadline, it is determined that the segmented pre-calculated latent variables have been successfully read within the first reading time budget; if the segmented pre-calculated latent variables have not been read by the first budget deadline, it is determined that the first reading timeout has occurred.
[0300] When the second-layer loading module 503 reads the long-interval pre-calculated latent variables covering the target original video frame interval through the second-layer loading path, it is further used for: The moment when the actual reading of the long interval pre-calculated latent variables is determined as the second reading start time; The time elapsed since the start of the second reading time budget is defined as the end time of the second budget. When the long-interval pre-calculated latent variables are read no later than the second budget deadline, it is determined that the long-interval pre-calculated latent variables have been successfully read within the second reading time budget; when the long-interval pre-calculated latent variables have not been read by the second budget deadline, it is determined that a second reading timeout has occurred.
[0301] In some optional implementations, before starting to read segmented pre-computed latent variables and before starting to read long-interval pre-computed latent variables, the first-level loading module 502 and the second-level loading module 503 are further configured to: Obtain the first delay sample set of the first layer loading path within the preset statistics window, and the second delay sample set of the second layer loading path within the preset statistics window. The first delay sample set and the second delay sample set respectively include the read delay of the recently successfully read for the corresponding loading path. Determine the first historical delay quantile value of the first delay sample set, and determine the first candidate time budget based on the first historical delay quantile value and the first time margin reserved for reading delay fluctuations; The first candidate time budget is limited to between the preset first minimum time budget and the preset first maximum time budget to obtain the first reading time budget; Determine the second historical delay quantile value of the second delay sample set, and determine the second candidate time budget based on the second historical delay quantile value and the second time margin reserved for reading delay fluctuations; The second candidate time budget is limited to between the preset second minimum time budget and the preset second maximum time budget to obtain the second reading time budget.
[0302] In some alternative implementations, the latent variable dynamic supply device 500 also includes an asynchronous preheating module 506.
[0303] After the third-level loading module 504 determines the online encoded latent variable as the target latent variable, the asynchronous warm-up module 506 is used for: Within the preset statistical window or since the start of training, the cumulative number of times the robot task trajectory data unit with trajectory unit identification has entered the third-level loading path for online encoding; Asynchronous preheating and submission judgment is performed only after the number of online coding attempts reaches the preset online coding attempt threshold; If the result of the asynchronous preheating submission judgment is met, and the preheating task submission conditions are met, after providing the target latent variable to the current training step, the background pre-computation task for generating the corresponding segmented pre-computation latent variable or long interval pre-computation latent variable is asynchronously submitted.
[0304] In some optional implementations, the asynchronous preheating module 506, when performing the asynchronous preheating submission judgment, is further configured to: Based on the sampling frequency of the corresponding robot task trajectory data unit and the expected number of remaining training steps, determine the expected number of times the robot task trajectory data unit will be accessed in the remaining training process. The expected net benefit of performing pre-computation is determined based on the expected number of visits, the single-pass latency savings between online encoding and pre-computed latent variable readings, and the equivalent cost required to perform pre-computation. If the expected net income meets the preset income conditions, the preheating task submission conditions are determined to be met; if the expected net income does not meet the preset income conditions, the preheating task submission conditions are determined not to be met.
[0305] When multiple background pre-computation tasks have been submitted, the asynchronous warm-up module 506 is further used for: The scheduling priority of each background pre-calculation task is determined based on the sampling frequency, expected delay saving per cycle, pre-calculation success probability, and pre-calculation equivalent cost of the robot task trajectory data unit corresponding to each background pre-calculation task. The execution order of multiple background pre-computation tasks is determined according to their scheduling priority from high to low.
[0306] The specific processing of each module, the meaning of related terms, optional implementation methods, and their technical effects in the device embodiments can be found in the descriptions of the corresponding steps in the foregoing method embodiments, and will not be repeated here. The above modules can be implemented by software, hardware, firmware, or a combination thereof, and can also be combined or further split according to the deployment method of computing and storage resources.
[0307] The following is for reference. Figure 6 , Figure 6 A schematic diagram of the structure of a computer system 600 suitable for implementing the electronic devices of the present application is shown. Figure 6 The computer system 600 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.
[0308] An electronic device, comprising one or more processors; Specifically, one or more processors can correspond to Figure 6The processing device 601 shown may include a central processing unit, a graphics processing unit, an artificial intelligence accelerator, a tensor processor, or other general-purpose or special-purpose processors. Different processors may respectively perform at least one of the following processing: training data request processing, pre-computation of latent variables, latent variable time-dimensional slicing, online encoding of video variational autoencoders, and provision of target latent variables.
[0309] A storage device on which one or more programs are stored; Specifically, the storage device may include at least one of read-only memory (ROM) 602, random access memory (RAM) 603, and storage device 608. ROM 602 may store a boot program or a firmware program; RAM 603 may store programs and data used during the operation of the electronic device; storage device 608 may include a disk, solid-state drive, flash memory device, or other non-volatile storage device, and may store one or more programs for implementing the latent variable dynamic supply method.
[0310] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the above embodiments.
[0311] Specifically, one or more programs may include program instructions for obtaining current training data requests, program instructions for reading pre-computed latent variables according to the first-layer loading path and the second-layer loading path, program instructions for performing latent variable time dimension slicing on long-interval pre-computed latent variables, program instructions for performing online encoding through the third-layer loading path, and program instructions for providing target latent variables to the current training step.
[0312] One or more programs may also include program instructions for determining a first recoverable failure and a second recoverable failure, performing configuration consistency checks and data integrity checks, determining a first read time budget and a second read time budget, accumulating the number of online encodings, performing asynchronous warm-up submission judgments, and scheduling background pre-computation tasks.
[0313] like Figure 6 As shown, the processing device 601, ROM 602, and RAM 603 can be connected to each other via bus 604. Input / output (I / O) interface 605 can also be connected to bus 604. When the electronic device is running, the processing device 601 can execute processing corresponding to the latent variable dynamic supply method according to a program stored in ROM 602 or a program loaded from storage device 608 into RAM 603.
[0314] I / O interface 605 can connect to input device 606, output device 607, storage device 608, and communication device 609. Input device 606 may include a keyboard, mouse, touch device, or other device for inputting device operating parameters. Output device 607 may include a display or other device for outputting device operating status.
[0315] The communication device 609 enables wired or wireless communication between electronic devices and training nodes, data storage nodes, object storage services, or other electronic devices. Electronic devices can receive current training data requests, read segmented pre-calculated latent variables, long-interval pre-calculated latent variables, or raw video frame sequences through the communication device 609, and provide target latent variables to the corresponding training nodes.
[0316] In some implementations, the processing device 601 may include multiple processors. One processor may perform training data loading and path switching, while another processor may perform online encoding of the video variational autoencoder. The multiple processors may share RAM 603, or each may have its own device memory and transfer target latent variables via high-speed interconnects.
[0317] In some implementations, the electronic device can be a single training server or a distributed electronic device consisting of multiple computing nodes. When the electronic device is a distributed electronic device, the request acquisition module, the first-layer loading module, the second-layer loading module, the third-layer loading module, and the latent variable providing module can be deployed on the same computing node or on different computing nodes that can communicate with each other.
[0318] Components in electronic devices are not required to all conform to Figure 6 The setup is as shown. Depending on the specific implementation requirements, the electronic device may omit some input or output devices, or it may add a graphics processor, artificial intelligence accelerator, direct memory access controller, high-speed network interface, or other components suitable for performing model training data processing.
[0319] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above embodiments.
[0320] Specifically, a computer-readable storage medium can be a tangible medium capable of containing or storing computer programs. Computer-readable storage media can include portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, flash memory, solid-state drives, optical storage devices, magnetic storage devices, or any combination of the above media.
[0321] The computer-readable storage medium may be disposed within the aforementioned electronic device or independently of the aforementioned electronic device. The computer program stored in the computer-readable storage medium may include program instructions for causing the processor to perform the processing corresponding to the first-level load path, the second-level load path, and the third-level load path, and may include program instructions for performing read time budget determination, configuration consistency verification, data integrity verification, and asynchronous warm-up processing.
[0322] When a computer program is executed by one or more processors, the one or more processors may perform all or some of the steps in the foregoing method embodiments. When executed jointly by multiple processors, different processors may respectively perform different processes in pre-computation latent variable reading, latent variable time dimension slicing, online encoding, and background pre-computation task scheduling.
[0323] A computer program product includes a computer program or computer instructions, which, when executed by a processor, implement any of the above embodiments.
[0324] Specifically, the computer program product may include computer programs or computer instructions for implementing a method for dynamically supplying latent variables. The computer program or computer instructions may be downloaded from a network and installed on an electronic device, or loaded onto an electronic device from a computer-readable storage medium. When executed by a processor, the computer program or computer instructions cause the electronic device to obtain the target latent variables sequentially according to the first-level loading path, the second-level loading path, and the third-level loading path, and provide the target latent variables to the current training step.
[0325] Computer programs or instructions can be written in one or more programming languages and can be executed entirely on a single electronic device, or separately on training nodes, data loading nodes, storage access nodes, and background pre-computation nodes. The program portions executing on different nodes can exchange current training data requests, pre-computed latent variables, raw video frame sequences, anomaly information, and target latent variables via wired or wireless networks.
[0326] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0327] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the module itself; for example, the first-level loading module can also be described as "a module that reads segmented pre-calculated latent variables through the first-level loading path".
[0328] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.
Claims
1. A method for dynamically supplying latent variables, characterized in that, include: Obtain the current training data request, wherein the current training data request includes the trajectory unit identifier and the target original video frame range; Based on the trajectory unit identifier and the target original video frame interval, the segmented pre-calculated latent variable corresponding to the target original video frame interval is read through the first layer loading path, and when the segmented pre-calculated latent variable is successfully read within the first reading time budget, the segmented pre-calculated latent variable is determined as the target latent variable; When the first recoverable failure or first read timeout occurs in the first layer loading path, switch to the second layer loading path, read the long interval pre-calculated latent variable covering the target original video frame interval through the second layer loading path, and when the long interval pre-calculated latent variable is successfully read within the second read time budget, perform latent variable time dimension slicing on the long interval pre-calculated latent variable to obtain the target latent variable; When the second layer loading path experiences a second recoverable failure or a second read timeout, the system switches to the third layer loading path, reads the original video frame sequence corresponding to the target original video frame interval, and performs online encoding on the original video frame sequence using the currently trained video variational autoencoder to obtain online encoding latent variables, and determines the online encoding latent variables as the target latent variables. The target latent variable is provided to the current training step.
2. The method according to claim 1, characterized in that, The step of reading segmented pre-calculated latent variables corresponding to the target original video frame interval through the first-layer loading path based on the trajectory unit identifier and the target original video frame interval includes: Obtain the first-level availability index, which is used to characterize the correspondence between the generated segmented pre-computed latent variables and the trajectory unit identifier, the effective original video frame interval, and the availability status of the segmented pre-computed latent variables; Based on the trajectory unit identifier and the target original video frame interval, query the first layer availability index to determine the segmented pre-calculated latent variables whose start and end boundaries of the valid original video frame interval are the same as the start and end boundaries of the target original video frame interval; When the first-level availability index indicates that the corresponding segment pre-calculated latent variable is in an available state, the corresponding segment pre-calculated latent variable is read through the first-level loading path.
3. The method according to claim 1, characterized in that, The step of reading long-range pre-computed latent variables covering the target original video frame range through the second-layer loading path, and performing latent variable time dimension slicing on the long-range pre-computed latent variables to obtain the target latent variables when the long-range pre-computed latent variables are successfully read within the second reading time budget, includes: Obtain the second-level availability index, which is used to characterize the correspondence between the generated long-interval pre-computed latent variables and the trajectory unit identifier, the effective original video frame interval, and the availability status of the long-interval pre-computed latent variables; Based on the trajectory unit identifier and the target original video frame interval, the second-level availability index is queried to determine the long interval pre-calculated latent variable that covers the target original video frame interval. The long interval pre-calculated latent variable is either a pre-calculated latent variable corresponding to the complete robot task trajectory data unit or a pre-calculated latent variable corresponding to a longer continuous original video frame interval that covers the target original video frame interval and has a coverage range greater than the target original video frame interval. When the second-level availability index indicates that the corresponding long-interval pre-calculated latent variable is in an available state, the corresponding long-interval pre-calculated latent variable is read through the second-level loading path; When the corresponding long interval pre-calculated latent variable is successfully read within the second reading time budget, the frame interval offset of the target original video frame interval relative to the starting point of the effective original video frame interval corresponding to the long interval pre-calculated latent variable is determined. Based on the frame interval offset, determine the latent variable time index range corresponding to the target original video frame interval; According to the time index range of the latent variables, the target latent variable is extracted from the long interval pre-calculated latent variables.
4. The method according to claim 3, characterized in that, The step of determining the latent variable time index range corresponding to the target original video frame interval based on the frame interval offset includes: Obtain the latent variable time grid corresponding to the long interval pre-calculated latent variable, and the latent variable time grid is used to characterize the correspondence between each latent variable time step in the long interval pre-calculated latent variable and the original video frame range; Based on the frame interval offset, determine the latent variable time step boundaries corresponding to the start and end boundaries of the target original video frame interval in the latent variable time grid, respectively. The start latent variable time index and the end latent variable time index are determined based on the latent variable time step boundaries corresponding to the start boundary and the end boundary, respectively. Based on the starting latent variable time index and the ending latent variable time index, the range of latent variable time indexes corresponding to the target original video frame interval is determined.
5. The method according to claim 1, characterized in that, The current training data request also includes the target camera set and the target camera channel order; Providing the target latent variable to the current training step includes: Obtain the camera channel information corresponding to the target latent variable obtained through the currently successfully loaded path; Based on the camera channel information, determine the camera channel corresponding to each target camera in the target camera set in the target latent variable; Select the latent variables of the camera channels corresponding to each target camera from the target latent variables, and arrange the selected latent variables according to the target camera channel order; The sorted latent variables are provided as the target latent variables to the current training step.
6. The method according to claim 1, characterized in that, The current training data request also includes encoding configuration information, which includes the video variational autoencoder version, temporal downsampling factor, and preprocessing version. The first recoverable failure includes at least one of the following: There are no segmented pre-calculated latent variables corresponding to the trajectory unit identifier and the target original video frame interval, or the corresponding segmented pre-calculated latent variables are in an unavailable state; The video variational autoencoder version, temporal downsampling factor, or preprocessing version corresponding to the pre-calculated latent variables in the corresponding segments are inconsistent with the video variational autoencoder version, temporal downsampling factor, or preprocessing version corresponding to the current training data request. A temporary network access exception, object storage rate limiting, or other temporary access exception occurred when reading the corresponding segmented pre-calculated latent variables. The second recoverable failure includes at least one of the following: There is no long interval pre-calculated latent variable that corresponds to the trajectory unit identifier and whose valid original video frame interval covers the target original video frame interval, or the corresponding long interval pre-calculated latent variable is in an unavailable state; The video variational autoencoder version, temporal downsampling factor, or preprocessing version corresponding to the pre-calculated latent variables over the corresponding long interval is inconsistent with the video variational autoencoder version, temporal downsampling factor, or preprocessing version corresponding to the current training data request. It is impossible to determine the latent variable time index range that can be used to perform latent variable time dimension slicing on the corresponding long interval based on the target original video frame interval; Temporary network access exceptions, object storage rate limiting, or other temporary access exceptions occurred when reading the corresponding long interval pre-calculated latent variables.
7. The method according to claim 6, characterized in that, Before determining the segmented pre-computed latent variables obtained through the first-layer loading path as the target latent variables, or before performing latent variable time dimension slicing on the long-interval pre-computed latent variables obtained through the second-layer loading path, the method further includes: Determine the request consistency field corresponding to the current training data request. The request consistency field includes at least one of the following configuration fields: trajectory unit identifier, target range, camera configuration information, encoding configuration information, video preprocessing information, and latent variable format information. The target range includes the target original video frame interval or its corresponding target latent variable time range. Extract the segmented consistency field corresponding to the request consistency field from the metadata of the segmented pre-computed latent variables, or determine the long interval consistency field corresponding to the part of the latent variables to be cut out based on the metadata of the long interval pre-computed latent variables. Following the same field order and deterministic serialization rules, a request configuration fingerprint is generated based on the request consistency field, and a segmented configuration fingerprint or a long interval configuration fingerprint is generated based on the segmented consistency field or the long interval consistency field. The request configuration fingerprint is compared with the segmented configuration fingerprint or the long interval configuration fingerprint, and target range coverage verification and data integrity verification are performed on the segmented pre-calculated latent variables or the long interval pre-calculated latent variables; When the configured fingerprint is consistent, covers the target range, and passes the data integrity verification, it is determined that the segmented pre-calculated latent variable meets the usage conditions as the target latent variable, or it is determined that the long interval pre-calculated latent variable meets the usage conditions for executing latent variable time dimension slicing. When the configured fingerprint is inconsistent or does not cover the target range, the first recoverable failure is determined to occur in the first layer loading path based on the segmented pre-calculated latent variables, or the second recoverable failure is determined to occur in the second layer loading path based on the long interval pre-calculated latent variables. When the segmented pre-calculated latent variable fails the data integrity check, the data integrity anomaly of the segmented pre-calculated latent variable is recorded and isolated, and the process is switched to the second-level loading path; when the long-interval pre-calculated latent variable fails the data integrity check, the data integrity anomaly of the long-interval pre-calculated latent variable is recorded and isolated, and the process is switched to the third-level loading path.
8. The method according to claim 1, characterized in that, When reading the segmented pre-calculated latent variables corresponding to the target original video frame interval through the first-level loading path, the method further includes: The moment when the actual reading of the segmented pre-calculated latent variables begins is determined as the first reading start moment; The time elapsed from the first reading start time to the first reading time budget is defined as the first budget cutoff time. When the segmented pre-calculated latent variable is read no later than the first budget deadline, it is determined that the segmented pre-calculated latent variable has been successfully read within the first reading time budget; when the segmented pre-calculated latent variable has not been read by the first budget deadline, it is determined that the first reading timeout has occurred. When reading long-interval pre-calculated latent variables covering the target original video frame interval through the second-layer loading path, the method further includes: The moment when the actual reading of the long interval pre-calculated latent variables is determined as the second reading start time; The time elapsed from the second reading start time to the second reading time budget is determined as the second budget cutoff time. When the long-interval pre-calculated latent variable is read no later than the second budget deadline, it is determined that the long-interval pre-calculated latent variable has been successfully read within the second reading time budget; when the long-interval pre-calculated latent variable has not been read by the second budget deadline, it is determined that the second reading timeout has occurred.
9. The method according to claim 8, characterized in that, Before starting to read the segmented pre-computed latent variables and before starting to read the long-interval pre-computed latent variables, the method further includes: Obtain a first delay sample set of the first layer loading path within a preset statistical window, and a second delay sample set of the second layer loading path within the preset statistical window. The first delay sample set and the second delay sample set respectively include the read delay of the corresponding loading path that was recently successfully read. Determine the first historical latency quantile value of the first latency sample set, and determine the first candidate time budget based on the first historical latency quantile value and the first time margin reserved for reading latency fluctuations; The first candidate time budget is limited to between a preset first minimum time budget and a preset first maximum time budget to obtain the first reading time budget; Determine the second historical latency quantile value of the second latency sample set, and determine the second candidate time budget based on the second historical latency quantile value and the second time margin reserved for reading latency fluctuations; The second candidate time budget is limited to between a preset second minimum time budget and a preset second maximum time budget to obtain the second reading time budget.
10. The method according to claim 1, characterized in that, After determining the online encoded latent variable as the target latent variable, the method further includes: Within a preset statistical window or since the start of training, the cumulative number of online encodings of robot task trajectory data units with the trajectory unit identifier entering the third-layer loading path; Asynchronous preheating submission judgment is performed only after the number of online encoding attempts reaches a preset online encoding attempt threshold; When the result of the asynchronous preheating submission judgment is met, and the preheating task submission conditions are met, after providing the target latent variable to the current training step, an asynchronous background pre-computation task is submitted to generate the corresponding segmented pre-computation latent variable or long interval pre-computation latent variable.
11. The method according to claim 10, characterized in that, The asynchronous preheating submission judgment includes: Based on the sampling frequency of the corresponding robot task trajectory data unit and the expected number of remaining training steps, the expected number of visits to the robot task trajectory data unit in the remaining training process is determined. The expected net benefit of performing pre-computation is determined based on the estimated number of accesses, the single-pass latency savings between online encoding and pre-computed latent variable readings, and the equivalent cost required to perform pre-computation. If the expected net revenue meets the preset revenue conditions, the preheating task submission conditions are determined to be met; if the expected net revenue does not meet the preset revenue conditions, the preheating task submission conditions are determined not to be met. When multiple submitted background pre-computation tasks exist, the method further includes: The scheduling priority of each background pre-calculation task is determined based on the sampling frequency, expected delay saving per cycle, pre-calculation success probability, and pre-calculation equivalent cost of the robot task trajectory data unit corresponding to each background pre-calculation task. The execution order of the multiple background pre-computation tasks is determined according to the scheduling priority from high to low.
12. A latent variable dynamic supply device, characterized in that, include: The request acquisition module is used to acquire the current training data request, which includes the trajectory unit identifier and the target original video frame range; The first-layer loading module is used to read the segmented pre-calculated latent variables corresponding to the target original video frame interval through the first-layer loading path according to the trajectory unit identifier and the target original video frame interval, and when the segmented pre-calculated latent variables are successfully read within the first reading time budget, the segmented pre-calculated latent variables are determined as target latent variables. The second-layer loading module is used to read long-range pre-calculated latent variables covering the target original video frame range through the second-layer loading path when the first recoverable failure or the first reading timeout occurs in the first-layer loading path, and to perform latent variable time dimension slicing on the long-range pre-calculated latent variables to obtain the target latent variables when the long-range pre-calculated latent variables are successfully read within the second reading time budget. The third-layer loading module is used to read the original video frame sequence corresponding to the target original video frame interval when the second-layer loading path experiences a second recoverable failure or a second read timeout, and to perform online encoding on the original video frame sequence using the currently trained video variational autoencoder to obtain online encoding latent variables, and to determine the online encoding latent variables as the target latent variables. The latent variable providing module is used to provide the target latent variable to the current training step.
13. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 11.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 11.
15. A computer program product, characterized in that, It includes a computer program or computer instructions, which, when executed by a processor, implement the method of any one of claims 1 to 11.
Citation Information
Patent Citations
Disturbance generation model training method and device
CN121236509A
Structural health monitoring abnormal data reconstruction method based on submerged space diffusion model
CN121834154A
