Video latent variable generation method and device and related product

Abstract

The application provides a video latent variable generation method and device and related products. The method comprises: acquiring robot task trajectory data; determining a task time sequence sensitivity level according to task attribute information, and determining a motion complexity level according to a video frame sequence and a robot action sequence; determining a video processing configuration according to the task time sequence sensitivity level and the motion complexity level, and determining a target time sequence segmentation interval that meets a time coding constraint of a video variational autoencoder; determining whether there is a reusable latent variable according to a latent variable reuse condition, and obtaining a target latent variable based on the reusable latent variable or video variational autoencoder coding. The application can adapt to the video processing needs of different robot tasks, and improve the efficiency and accuracy of video latent variable generation.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method, apparatus, and related products for generating latent variables in videos. Background Technology

[0002] With the development of artificial intelligence technology, Vision-Language-Action Models (VLA Models) are increasingly being applied to the field of robot task learning. VLA Models typically require training using task trajectory data collected during robot task execution. To reduce the complexity of video data processing during VLA Model training, video coding models can be used to encode the video information in the task trajectory data, converting it into a low-dimensional latent variable representation for subsequent model training.

[0003] However, existing methods for generating latent variables in videos still have the following problems: On the one hand, existing video coding schemes typically use fixed video processing parameters to encode video information in different task data, without considering the differences in video content change characteristics and motion change characteristics corresponding to different robot tasks. This makes it difficult for fixed video processing methods to adapt to the video representation requirements of different task data.

[0004] On the other hand, video coding models usually have coding constraints in the time dimension. Existing video segmentation methods may result in some video segments having insufficient video frames to meet the time constraints in the video coding process, thus affecting the video latent variable generation process.

[0005] Therefore, it is necessary to propose a video latent variable generation method to solve at least one of the above-mentioned technical problems. Summary of the Invention

[0006] This application proposes a video latent variable generation method, apparatus, and related products. These methods adaptively determine video processing configurations based on the characteristics of robot task trajectory data and generate video latent variables that satisfy video encoding time constraints, thereby improving the adaptability and effectiveness of the video latent variable generation process. Furthermore, by reusing existing latent variables, target latent variables are generated using existing latent variables when conditions are met, reducing redundant encoding processes and improving the efficiency of video latent variable generation.

[0007] This application provides a method for generating latent variables in video, the method comprising: Acquire robot task trajectory data, which includes video frame sequences, robot action sequences, and task attribute information; The task timing sensitivity level of the robot task trajectory data is determined based on the task attribute information. The motion complexity level of the robot task trajectory data is determined based on the video frame sequence and the robot motion sequence. The video processing configuration is determined based on the task temporal sensitivity level and the motion complexity level; wherein, the video processing configuration includes target motion configuration parameters and the target temporal downsampling factor corresponding to the video variational autoencoder; According to the video processing configuration, multiple target temporal segmentation intervals that satisfy the temporal coding constraints of the video variational autoencoder are determined in the video frame sequence; wherein, the temporal coding constraints of the video variational autoencoder include the temporal alignment constraint between the number of input video frames and the target temporal downsampling factor; For any of the target time series segment intervals, determine whether there are reusable latent variables corresponding to the target time series segment intervals based on the latent variable reuse conditions, and determine the target latent variables corresponding to the target time series segment intervals based on the determination results; When there is a reusable latent variable that satisfies the latent variable reuse condition, the target latent variable for the target time series segment interval is obtained based on the reusable latent variable; When there is no reusable latent variable that satisfies the latent variable reuse condition, video variational autoencoder encoding is performed on the video frame sequence of the target temporal segment interval to obtain the target latent variable of the target temporal segment interval.

[0008] In some optional implementations, determining the motion complexity level corresponding to the robot task trajectory data based on the video frame sequence and the robot motion sequence includes: Based on the video content change information between adjacent video frames in the video frame sequence, video motion features of the video frame sequence are generated; wherein, the video motion features are used to characterize the overall video motion level, the degree of video motion change, and the peak video motion of the video frame sequence. Based on the motion change information between adjacent robot actions in the robot motion sequence, motion motion features of the robot motion sequence are generated; wherein, the motion motion features are used to characterize the overall motion change level, motion change degree, and motion change peak of the robot motion sequence. Based on the video motion features and the action motion features, determine the motion complexity corresponding to the robot task trajectory data; Obtain a preset threshold for classifying motion complexity levels, and determine the motion complexity level of the robot task trajectory data based on the threshold for classifying motion complexity levels.

[0009] In some optional implementations, the task attribute information includes at least one of robot state machine stage information, task tags, skill identifiers, and language instruction information. Determining the task temporal sensitivity level of the robot task trajectory data based on the task attribute information includes: Obtain a pre-defined first correspondence table representing the relationship between a set of task time sensitivity levels and at least one of robot state machine stage information, task tags, and skill identifiers; wherein the set of task time sensitivity levels includes high time sensitivity level, medium time sensitivity level, and low time sensitivity level; When the task attribute information includes at least one of robot state machine stage information, task tag or skill identifier, the task time sensitivity level corresponding to the robot task trajectory data is determined according to the first correspondence table. When the task attribute information does not include robot state machine stage information, task tags, and skill identifiers, task feature recognition is performed based on the language instruction information, and the task timing sensitivity level corresponding to the robot task trajectory data is determined based on the recognized task features.

[0010] In some optional implementations, determining the video processing configuration based on the task timing sensitivity level and the motion complexity level includes: Obtain a preset set of candidate action configuration parameters and a set of candidate time-series downsampling factors; wherein, the set of candidate action configuration parameters includes at least one candidate action configuration parameter, the candidate action configuration parameter includes the number of historical observation steps, the number of action prediction steps, and the observation sampling span, and the set of candidate time-series downsampling factors includes at least one candidate time-series downsampling factor; Based on the task timing sensitivity level and the motion complexity level, the target timing downsampling factor is determined from the candidate timing downsampling factor set, and the target action configuration parameter is determined from the candidate action configuration parameter set.

[0011] In some optional implementations, determining the target temporal downsampling factor from the candidate temporal downsampling factor set based on the task temporal sensitivity level and the motion complexity level, and determining the target action configuration parameter from the candidate action configuration parameter set, includes: The target temporal downsampling factor is determined from the set of candidate temporal downsampling factors according to a pre-set second correspondence table used to characterize the correspondence between the task temporal sensitivity level, the motion complexity level and each candidate temporal downsampling factor; Based on a pre-defined third correspondence table characterizing the relationship between the task's temporal sensitivity level, the motion complexity level, and each candidate action configuration parameter, the target action configuration parameter is determined from the set of candidate action configuration parameters; or, Input the task timing sensitivity level and / or the motion complexity level into a preset configuration selection model to obtain the selection probability corresponding to each candidate action configuration parameter; determine the candidate action configuration parameter with the highest selection probability based on the selection probability corresponding to each candidate action configuration parameter; and determine the candidate action configuration parameter with the highest selection probability as the target action configuration parameter.

[0012] In some optional implementations, determining multiple target temporal segmentation intervals in the video frame sequence that satisfy the temporal coding constraints of the video variational autoencoder, according to the video processing configuration, includes: The temporal segmentation interval length of the video frame sequence is determined based on the target action configuration parameters and the target temporal downsampling factor; The video frame sequence is divided according to the length of the time segmentation interval to obtain multiple initial time segmentation intervals; Determine whether there are any initial temporal segmentation intervals among the plurality of initial temporal segmentation intervals that do not satisfy the temporal coding constraints of the video variational autoencoder; If not, the plurality of initial temporal segmentation intervals are determined as the plurality of target temporal segmentation intervals that satisfy the temporal coding constraints of the video variational autoencoder; If it exists, the initial temporal segment interval that does not meet the temporal coding constraints of the video variational autoencoder is adjusted according to the temporal segment interval length to obtain the adjusted initial temporal segment interval, wherein the adjusted initial temporal segment interval meets the temporal coding constraints of the video variational autoencoder. The initial temporal segmentation interval that satisfies the temporal coding constraints of the video variational autoencoder and the adjusted initial temporal segmentation interval are determined as the plurality of target temporal segmentation intervals that satisfy the temporal coding constraints of the video variational autoencoder.

[0013] In some optional implementations, adjusting the initial temporal segmentation intervals that do not meet the temporal coding constraints of the video variational autoencoder based on the length of the temporal segmentation intervals to obtain adjusted initial temporal segmentation intervals includes: The number of supplementary video frames to be added is determined based on the length of the time segment interval. Generate supplementary video frames based on the number of supplementary video frames; The supplementary video frame is added to the initial temporal segmentation interval that does not meet the temporal coding constraints of the video variational autoencoder to obtain the adjusted initial temporal segmentation interval, wherein the adjusted initial temporal segmentation interval includes the original video frame and the supplementary video frame.

[0014] In some optional implementations, determining whether a reusable latent variable exists based on the latent variable reuse condition includes: Query at least one existing valid latent variable that has been encoded by a variational autoencoder for video; Determine the parent time series segment interval for each of the existing effective latent variables; Determine whether there exists a reusable parent time series segment interval that satisfies the latent variable reuse condition; When the reusable parent time series segmentation interval exists, it is determined that there is a reusable latent variable corresponding to the target time series segmentation interval; When the reusable parent time series segmentation interval does not exist, it is determined that there is no reusable latent variable corresponding to the target time series segmentation interval.

[0015] In some optional implementations, the latent variable reuse conditions include at least one of the following: time interval coverage condition, time base consistency condition, time index mapping condition, time-series coding condition, context sufficiency condition, and coding condition consistency condition; wherein, The time interval coverage condition is used to determine whether the parent time segment interval covers the target time segment interval; The time reference consistency condition is used to determine whether the parent time segment interval and the target time segment interval are determined based on the same original video frame time axis. The time index mapping condition is used to determine whether the offset of the starting position of the target time segment interval relative to the parent time segment interval meets the time index mapping requirement of the target time downsampling factor; The timing coding conditions are used to determine whether the parent timing segmentation interval and the target timing segmentation interval use the same video variational autoencoder timing downsampling factor; The context sufficiency condition is used to determine whether the parent temporal segmentation interval can provide context information that satisfies the temporal coding requirements of the video variational autoencoder for the target temporal segmentation interval; The coding condition consistency condition is used to determine whether the coding conditions of the video variational autoencoder corresponding to the parent temporal segment interval and the target temporal segment interval are consistent.

[0016] In some optional implementations, obtaining the target latent variable of the target time series segment interval based on the reusable latent variable includes: Based on the time offset of the target time series segment interval relative to the reusable parent time series segment interval, determine the reusable latent variable time index range corresponding to the reusable latent variable; Based on the reusable latent variable time index range, time slicing is performed on the reusable latent variable to obtain the target latent variable in the target time series segment interval.

[0017] In some alternative implementations, The step of performing video variational autoencoder encoding on the video frame sequence of the target temporal segmentation interval to obtain the target latent variable of the target temporal segmentation interval includes: Obtain the effective temporal receptive field range of the video variational autoencoder; Based on the effective temporal receptive field range, context video frames are added to the left and / or right of the target temporal segmentation interval to generate an extended video frame sequence, and the context video frame index range corresponding to the context video frames in the extended video frame sequence is determined. The extended video frame sequence is encoded using a video variational autoencoder to obtain the initial latent variables of the target temporal segmentation interval; Based on the context video frame index range and the target temporal downsampling factor, determine the context latent variable corresponding to the context video frame in the initial latent variable, and remove the context latent variable; The remaining latent variables after removing the context latent variables are determined as the target latent variables of the target time series segment interval.

[0018] In some optional implementations, after determining multiple target temporal segmentation intervals in the video frame sequence that satisfy the temporal coding constraints of the video variational autoencoder according to the video processing configuration, the method further includes: For any of the target time-series segmentation intervals, within a preset boundary range, multiple candidate boundary adjustment positions are determined; Based on the video frame timestamps corresponding to each candidate boundary adjustment position and the action timestamps corresponding to each robot action in the robot action sequence, determine the target action index corresponding to each candidate boundary adjustment position; based on the target action index corresponding to each candidate boundary adjustment position, determine the robot action data corresponding to each target action index; Based on the robot motion data, determine the first boundary evaluation result corresponding to each candidate boundary adjustment position; The candidate boundary adjustment position with the smallest first boundary evaluation result is determined as the target boundary adjustment position; Based on the positional difference between the target boundary adjustment position and the current boundary position of the target temporal segment interval, the boundary translation amount is determined. The boundary translation amount is used to represent the translation direction and translation distance of the target temporal segment interval in the video frame sequence. According to the boundary translation amount, the starting boundary and ending boundary of the target temporal segment interval are synchronously translated in the same direction and by the same distance to obtain the first target temporal segment interval after translation.

[0019] In some alternative implementations, When multiple target temporal segmentation intervals are used as independent training samples, after determining multiple target temporal segmentation intervals that satisfy the temporal coding constraints of the video variational autoencoder in the video frame sequence according to the video processing configuration, the method further includes: For any of the target time series segmentation intervals, multiple valid candidate boundary shifts are determined; wherein each valid candidate boundary shift is an integer multiple of the target time series downsampling factor; Based on multiple legitimate candidate boundary translations, the starting and ending boundaries of the target time series segment intervals are synchronously translated to obtain multiple candidate time series segment intervals that correspond one-to-one with each legitimate candidate boundary translation. Determine the second boundary evaluation results for each candidate time series segment interval; The candidate time series segment interval with the smallest second boundary evaluation result is determined as the second target time series segment interval after translation.

[0020] In some optional implementations, when a continuous temporal segment sequence consisting of multiple target temporal segment intervals exists within the multiple target temporal segment intervals, after determining multiple target temporal segment intervals satisfying the video variational autoencoder temporal coding constraints in the video frame sequence according to the video processing configuration, the method further includes: Determine the range of video frame indices for the segmented sequence corresponding to the continuous temporal segmented sequence; Multiple candidate global segment boundary translation amounts are determined based on the segmented sequence video frame index range; Based on the preset legal constraints of global segment boundary translation, the multiple candidate global segment boundary translations are filtered to obtain legal candidate global segment boundary translations. Based on the translation amount of each legal candidate global segment boundary, the segment boundary corresponding to the continuous temporal segment sequence is translated to obtain the corresponding candidate continuous temporal segment sequence. Determine the global segmentation boundary evaluation results for each of the candidate continuous temporal segmentation sequences; The candidate continuous temporal segment sequence with the smallest global segment boundary evaluation result is determined as the translated continuous temporal segment sequence.

[0021] In some optional implementations, the method further includes: For the same robot task trajectory data, coarse-grained time-series segmentation intervals and fine-grained time-series segmentation intervals are established according to different time granularities. The coarse-grained time-series segmentation video frame index range corresponding to the coarse-grained time-series segmentation interval is larger than the fine-grained time-series segmentation video frame range corresponding to the fine-grained time-series segmentation interval. When the fine-grained time series segmentation interval is completely covered by a single coarse-grained time series segmentation interval, the fine-grained time series segmentation latent variable corresponding to the fine-grained time series segmentation interval is determined according to the coarse-grained time series segmentation latent variable corresponding to the coarse-grained time series segmentation interval. When the fine-grained temporal segmentation interval crosses the boundary position between multiple coarse-grained temporal segmentation intervals, video variational autoencoder encoding is performed on the fine-grained temporal segmentation interval to obtain the fine-grained temporal segmentation latent variable corresponding to the fine-grained temporal segmentation interval.

[0022] In some alternative implementations, after obtaining the target latent variable, the method further includes: Save the target latent variables and the corresponding video processing configuration, and use the target latent variables for training the vision-language-action model.

[0023] Secondly, this application provides a video latent variable generation apparatus, comprising: The task trajectory acquisition module is used to acquire robot task trajectory data, which includes video frame sequences, robot action sequences, and task attribute information. The task timing sensitivity level determination module is used to determine the task timing sensitivity level of the robot task trajectory data based on the task attribute information. A motion complexity level determination module is used to determine the motion complexity level of the robot task trajectory data based on the video frame sequence and the robot action sequence; A video processing configuration determination module is used to determine a video processing configuration based on the task temporal sensitivity level and the motion complexity level; wherein, the video processing configuration includes target motion configuration parameters and a target temporal downsampling factor corresponding to the video variational autoencoder; The temporal segmentation interval determination module is used to determine, according to the video processing configuration, multiple target temporal segmentation intervals in the video frame sequence that satisfy the temporal coding constraints of the video variational autoencoder; wherein, the temporal coding constraints of the video variational autoencoder include a time alignment constraint between the number of input video frames and the target temporal downsampling factor; The latent variable reuse judgment module is used to determine, for any target time series segment interval, whether there is a reusable latent variable corresponding to the target time series segment interval according to the latent variable reuse condition, and to determine the target latent variable corresponding to the target time series segment interval according to the judgment result; The latent variable reuse module is used to obtain the target latent variable of the target time series segment interval based on the reusable latent variable when there is a reusable latent variable that satisfies the latent variable reuse condition; The latent variable encoding module is used to perform video variational autoencoder encoding on the video frame sequence of the target temporal segmentation interval when there is no reusable latent variable that satisfies the latent variable reuse condition, so as to obtain the target latent variable of the target temporal segmentation interval.

[0024] In some optional implementations, the motion complexity level determination module is further configured to: Based on the video content change information between adjacent video frames in the video frame sequence, video motion features of the video frame sequence are generated; wherein, the video motion features are used to characterize the overall video motion level, the degree of video motion change, and the peak video motion of the video frame sequence. Based on the motion change information between adjacent robot actions in the robot motion sequence, motion motion features of the robot motion sequence are generated; wherein, the motion motion features are used to characterize the overall motion change level, motion change degree, and motion change peak of the robot motion sequence. Based on the video motion features and the action motion features, determine the motion complexity corresponding to the robot task trajectory data; Obtain a preset threshold for classifying motion complexity levels, and determine the motion complexity level of the robot task trajectory data based on the threshold for classifying motion complexity levels.

[0025] In some optional implementations, the task attribute information includes at least one of robot state machine stage information, task tags, skill identifiers, and language instruction information, and the task timing sensitivity level determination module is further used for: Obtain a pre-defined first correspondence table representing the relationship between a set of task time sensitivity levels and at least one of robot state machine stage information, task tags, and skill identifiers; wherein the set of task time sensitivity levels includes high time sensitivity level, medium time sensitivity level, and low time sensitivity level; When the task attribute information includes at least one of robot state machine stage information, task tag or skill identifier, the task time sensitivity level corresponding to the robot task trajectory data is determined according to the first correspondence table. When the task attribute information does not include robot state machine stage information, task tags, and skill identifiers, task feature recognition is performed based on the language instruction information, and the task timing sensitivity level corresponding to the robot task trajectory data is determined based on the recognized task features.

[0026] In some optional implementations, the video processing configuration determination module is further configured to: Obtain a preset set of candidate action configuration parameters and a set of candidate time-series downsampling factors; wherein, the set of candidate action configuration parameters includes at least one candidate action configuration parameter, the candidate action configuration parameter includes the number of historical observation steps, the number of action prediction steps, and the observation sampling span, and the set of candidate time-series downsampling factors includes at least one candidate time-series downsampling factor; Based on the task timing sensitivity level and the motion complexity level, the target timing downsampling factor is determined from the candidate timing downsampling factor set, and the target action configuration parameter is determined from the candidate action configuration parameter set.

[0027] In some optional implementations, the video processing configuration determination module is further configured to: The target temporal downsampling factor is determined from the set of candidate temporal downsampling factors according to a pre-set second correspondence table used to characterize the correspondence between the task temporal sensitivity level, the motion complexity level and each candidate temporal downsampling factor; Based on a pre-defined third correspondence table characterizing the relationship between the task's temporal sensitivity level, the motion complexity level, and each candidate action configuration parameter, the target action configuration parameter is determined from the set of candidate action configuration parameters; or, Input the task timing sensitivity level and / or the motion complexity level into a preset configuration selection model to obtain the selection probability corresponding to each candidate action configuration parameter; determine the candidate action configuration parameter with the highest selection probability based on the selection probability corresponding to each candidate action configuration parameter; and determine the candidate action configuration parameter with the highest selection probability as the target action configuration parameter.

[0028] In some optional implementations, the time-series segmentation interval determination module is further configured to: The temporal segmentation interval length of the video frame sequence is determined based on the target action configuration parameters and the target temporal downsampling factor; The video frame sequence is divided according to the length of the time segmentation interval to obtain multiple initial time segmentation intervals; Determine whether there are any initial temporal segmentation intervals among the plurality of initial temporal segmentation intervals that do not satisfy the temporal coding constraints of the video variational autoencoder; If not, the plurality of initial temporal segmentation intervals are determined as the plurality of target temporal segmentation intervals that satisfy the temporal coding constraints of the video variational autoencoder; If it exists, the initial temporal segment interval that does not meet the temporal coding constraints of the video variational autoencoder is adjusted according to the temporal segment interval length to obtain the adjusted initial temporal segment interval, wherein the adjusted initial temporal segment interval meets the temporal coding constraints of the video variational autoencoder. The initial temporal segmentation interval that satisfies the temporal coding constraints of the video variational autoencoder and the adjusted initial temporal segmentation interval are determined as the plurality of target temporal segmentation intervals that satisfy the temporal coding constraints of the video variational autoencoder.

[0029] In some optional implementations, the time-series segmentation interval determination module is further configured to: The number of supplementary video frames to be added is determined based on the length of the time segment interval. Generate supplementary video frames based on the number of supplementary video frames; The supplementary video frame is added to the initial temporal segmentation interval that does not meet the temporal coding constraints of the video variational autoencoder to obtain the adjusted initial temporal segmentation interval, wherein the adjusted initial temporal segmentation interval includes the original video frame and the supplementary video frame.

[0030] In some optional implementations, the latent variable reuse determination module is further configured to: Query at least one existing valid latent variable that has been encoded by a variational autoencoder for video; Determine the parent time series segment interval for each of the existing effective latent variables; Determine whether there exists a reusable parent time series segment interval that satisfies the latent variable reuse condition; When the reusable parent time series segmentation interval exists, it is determined that there is a reusable latent variable corresponding to the target time series segmentation interval; When the reusable parent time series segmentation interval does not exist, it is determined that there is no reusable latent variable corresponding to the target time series segmentation interval.

[0031] In some optional implementations, the latent variable reuse conditions include at least one of the following: time interval coverage condition, time base consistency condition, time index mapping condition, time-series coding condition, context sufficiency condition, and coding condition consistency condition; wherein, The time interval coverage condition is used to determine whether the parent time segment interval covers the target time segment interval; The time reference consistency condition is used to determine whether the parent time segment interval and the target time segment interval are determined based on the same original video frame time axis. The time index mapping condition is used to determine whether the offset of the starting position of the target time segment interval relative to the parent time segment interval meets the time index mapping requirement of the target time downsampling factor; The timing coding conditions are used to determine whether the parent timing segmentation interval and the target timing segmentation interval use the same video variational autoencoder timing downsampling factor; The context sufficiency condition is used to determine whether the parent temporal segmentation interval can provide context information that satisfies the temporal coding requirements of the video variational autoencoder for the target temporal segmentation interval; The coding condition consistency condition is used to determine whether the coding conditions of the video variational autoencoder corresponding to the parent temporal segment interval and the target temporal segment interval are consistent.

[0032] In some optional implementations, the latent variable multiplexing module is further configured to: Based on the time offset of the target time series segment interval relative to the reusable parent time series segment interval, determine the reusable latent variable time index range corresponding to the reusable latent variable; Based on the reusable latent variable time index range, time slicing is performed on the reusable latent variable to obtain the target latent variable in the target time series segment interval.

[0033] In some optional implementations, the latent variable encoding module is further configured to: Obtain the effective temporal receptive field range of the video variational autoencoder; Based on the effective temporal receptive field range, context video frames are added to the left and / or right of the target temporal segmentation interval to generate an extended video frame sequence, and the context video frame index range corresponding to the context video frames in the extended video frame sequence is determined. The extended video frame sequence is encoded using a video variational autoencoder to obtain the initial latent variables of the target temporal segmentation interval; Based on the context video frame index range and the target temporal downsampling factor, determine the context latent variable corresponding to the context video frame in the initial latent variable, and remove the context latent variable; The remaining latent variables after removing the context latent variables are determined as the target latent variables of the target time series segment interval.

[0034] In some optional embodiments, the apparatus further includes a boundary adjustment module, the boundary adjustment module being used for: For any of the target time-series segmentation intervals, within a preset boundary range, multiple candidate boundary adjustment positions are determined; Based on the video frame timestamps corresponding to each candidate boundary adjustment position and the action timestamps corresponding to each robot action in the robot action sequence, determine the target action index corresponding to each candidate boundary adjustment position; based on the target action index corresponding to each candidate boundary adjustment position, determine the robot action data corresponding to each target action index; Based on the robot motion data, determine the first boundary evaluation result corresponding to each candidate boundary adjustment position; The candidate boundary adjustment position with the smallest first boundary evaluation result is determined as the target boundary adjustment position; Based on the positional difference between the target boundary adjustment position and the current boundary position of the target temporal segment interval, the boundary translation amount is determined. The boundary translation amount is used to represent the translation direction and translation distance of the target temporal segment interval in the video frame sequence. According to the boundary translation amount, the starting boundary and ending boundary of the target temporal segment interval are synchronously translated in the same direction and by the same distance to obtain the first target temporal segment interval after translation.

[0035] In some optional implementations, when multiple target time-series segment intervals are used as independent training samples, the apparatus further includes an independent sample boundary optimization module, which is used to: For any of the target time series segmentation intervals, multiple valid candidate boundary shifts are determined; wherein each valid candidate boundary shift is an integer multiple of the target time series downsampling factor; Based on multiple legitimate candidate boundary translations, the starting and ending boundaries of the target time series segment intervals are synchronously translated to obtain multiple candidate time series segment intervals that correspond one-to-one with each legitimate candidate boundary translation. Determine the second boundary evaluation results for each candidate time series segment interval; The candidate time series segment interval with the smallest second boundary evaluation result is determined as the second target time series segment interval after translation.

[0036] In some optional embodiments, when a continuous time series segment sequence consisting of multiple target time series segment intervals exists within the multiple target time series segment intervals, the apparatus further includes a continuous segment global optimization module, the continuous segment global optimization module being used for: Determine the range of video frame indices for the segmented sequence corresponding to the continuous temporal segmented sequence; Multiple candidate global segment boundary translation amounts are determined based on the segmented sequence video frame index range; Based on the preset legal constraints of global segment boundary translation, the multiple candidate global segment boundary translations are filtered to obtain legal candidate global segment boundary translations. Based on the translation amount of each legal candidate global segment boundary, the segment boundary corresponding to the continuous temporal segment sequence is translated to obtain the corresponding candidate continuous temporal segment sequence. Determine the global segmentation boundary evaluation results for each of the candidate continuous temporal segmentation sequences; The candidate continuous temporal segment sequence with the smallest global segment boundary evaluation result is determined as the translated continuous temporal segment sequence.

[0037] In some optional embodiments, the apparatus further includes a multi-granularity latent variable multiplexing module, the multi-granularity latent variable multiplexing module being used for: For the same robot task trajectory data, coarse-grained time-series segmentation intervals and fine-grained time-series segmentation intervals are established according to different time granularities. The coarse-grained time-series segmentation video frame index range corresponding to the coarse-grained time-series segmentation interval is larger than the fine-grained time-series segmentation video frame range corresponding to the fine-grained time-series segmentation interval. When the fine-grained time series segmentation interval is completely covered by a single coarse-grained time series segmentation interval, the fine-grained time series segmentation latent variable corresponding to the fine-grained time series segmentation interval is determined according to the coarse-grained time series segmentation latent variable corresponding to the coarse-grained time series segmentation interval. When the fine-grained temporal segmentation interval crosses the boundary position between multiple coarse-grained temporal segmentation intervals, video variational autoencoder encoding is performed on the fine-grained temporal segmentation interval to obtain the fine-grained temporal segmentation latent variable corresponding to the fine-grained temporal segmentation interval.

[0038] In some optional embodiments, the apparatus further includes a model training application module, the model training application module being configured to: After obtaining the target latent variables, the target latent variables and the corresponding video processing configuration are saved, and the target latent variables are used for training the vision-language-action model.

[0039] Thirdly, this application provides an electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the above-mentioned one or more programs are executed by the above-mentioned one or more processors, the above-mentioned one or more processors implement the method as described in any embodiment of the first aspect of this application.

[0040] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method described in any embodiment of the first aspect of this application.

[0041] Fifthly, this application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the method described in any embodiment of the first aspect of this application.

[0042] The video latent variable generation method, apparatus, and related products provided in the embodiments of this application acquire robot task trajectory data including video frame sequences, robot motion sequences, and task attribute information. They determine the task temporal sensitivity level based on the task attribute information, determine the motion complexity level based on the video frame sequences and robot motion sequences, and then determine the video processing configuration based on the task temporal sensitivity level and motion complexity level. This allows the video processing process to be adjusted in conjunction with the task characteristics and motion change characteristics of the robot task data itself, thereby improving the adaptability of the video latent variable generation process to different robot task trajectory data.

[0043] Furthermore, by determining multiple target temporal segmentation intervals that satisfy the temporal coding constraints of the video variational autoencoder in the video frame sequence according to the video processing configuration, the number of video frames corresponding to the target temporal segmentation intervals can meet the temporal coding requirements of the video variational autoencoder, thereby improving the stability of the correspondence between the video temporal range and the temporal dimension of the video latent variables.

[0044] Furthermore, for any target temporal segmentation interval, the existence of reusable latent variables is determined based on the latent variable reuse condition. If a reusable latent variable that meets the latent variable reuse condition exists, the target latent variable is obtained based on the reusable latent variable. If no reusable latent variable that meets the latent variable reuse condition exists, a video variational autoencoder is executed to generate the target latent variable. This allows existing video latent variables to be effectively utilized when the reuse condition is met, reducing the repeated encoding process and improving the efficiency of video latent variable generation.

[0045] In summary, the technical solution provided in this application can determine the appropriate video processing configuration based on the characteristics of robot task trajectory data, generate target latent variables that meet the temporal coding constraints of the video variational autoencoder, and simultaneously enable the reuse of latent variables that meet the conditions, thereby improving the effectiveness of the video latent variable generation process. Attached Figure Description

[0046] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings: Figure 1 This is a system architecture diagram in which an embodiment of the video latent variable generation method according to this application can be applied; Figure 2This is a flowchart of an embodiment of the video latent variable generation method according to this application; Figure 3 This is an exploded flowchart of an embodiment of step S203 in the video latent variable generation method according to this application; Figure 4 This is a breakdown flowchart of an embodiment of step S204 in the video latent variable generation method of this application; Figure 5 This is an exploded flowchart of an embodiment of step S205 in the video latent variable generation method according to this application; Figure 6 This is an exploded flowchart of an embodiment of step S206 in the video latent variable generation method according to this application; Figure 7 This is a breakdown flowchart of an embodiment of step S208 in the video latent variable generation method according to this application; Figure 8 This is a schematic diagram of a structure of an embodiment of the video latent variable generation apparatus according to this application; Figure 9 This is a schematic diagram of the structure of an electronic device suitable for implementing embodiments of this application. Detailed Implementation

[0047] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not intended to limit the scope of the present application. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present application are shown in the accompanying drawings.

[0048] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0049] Figure 1 An exemplary system architecture 100 to which the video latent variable generation method of this application can be applied is shown.

[0050] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, network 104, and server 105. Network 104 is used to provide communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various communication connection types, such as wired communication links, wireless communication links, etc.

[0051] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be equipped with applications for robot task data processing, model training, or data management, such as robot control applications, data acquisition applications, model training management applications, and data processing applications.

[0052] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with data processing capabilities, including but not limited to intelligent robot control equipment, industrial control equipment, computer equipment, server terminals, portable computing devices, etc. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software modules or as a single software module. No specific limitations are made here.

[0053] Terminal devices 101, 102, and 103 can be used to acquire robot task trajectory data and can send the acquired data to server 105 for processing.

[0054] Server 105 can be a server that provides various data processing services. For example, server 105 can receive robot task trajectory data sent by terminal devices 101, 102, and 103, process the robot task trajectory data, and execute the video latent variable generation method provided in this application embodiment to generate corresponding video latent variables.

[0055] Specifically, server 105 can determine the task temporal sensitivity level corresponding to the robot task trajectory data based on task attribute information, determine the motion complexity level based on video frame sequence and robot action sequence, and determine the video processing configuration based on the task temporal sensitivity level and motion complexity level; further, it can determine the target temporal segmentation interval that satisfies the temporal coding constraints of video variational autoencoder based on the video processing configuration, and determine the corresponding target latent variable based on the latent variable reuse condition.

[0056] In some implementations, server 105 may also send the generated target latent variables to terminal devices 101, 102, 103, or to a vision-language-action model training system for subsequent vision-language-action model training.

[0057] In some cases, the video latent variable generation method provided in this application can be jointly executed by terminal devices 101, 102, and 103 and server 105. For example, terminal devices 101, 102, and 103 can be used to collect robot task trajectory data, and server 105 can be used to execute the video latent variable generation process based on the robot task trajectory data. This application does not limit this.

[0058] In some cases, the video latent variable generation method provided in this application can be executed by server 105. Correspondingly, the video latent variable generation device can also be set in server 105. In this case, the system architecture 100 may not include terminal devices 101, 102, and 103.

[0059] In some cases, the video latent variable generation method provided in this application can be executed by terminal devices 101, 102, and 103. Correspondingly, the video latent variable generation device can also be set in terminal devices 101, 102, and 103. In this case, the system architecture 100 may not include server 105.

[0060] It should be noted that server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules, or as a single software program or software module. No specific limitations are made here.

[0061] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0062] The video frame sequences, robot motion sequences, and task attribute information involved in this application can all be derived from data collection processes authorized by users or fully authorized by relevant parties, and the collection, use, and processing of related data all comply with relevant laws, regulations, and standards.

[0063] Continue to refer to Figure 2 , Figure 2 A flowchart of one embodiment of the video latent variable generation method according to this application is shown. Figure 2 The video latent variable generation method shown can be applied to Figure 1 The terminal device or server shown. This process S200 includes at least the following steps S201-S208.

[0064] Step S201: Obtain robot task trajectory data.

[0065] The video latent variable generation method of this application can be applied to the training data processing scenario of vision-language-action model.

[0066] Robot task trajectory data can be a collection of data organized in units of a robot performing a single task. In other words, a robot task trajectory data can correspond to the continuous data records generated during the robot's completion of a complete task.

[0067] For example, data collected during the robot's performance of object grasping, object placement, assembly operations, or other interactive tasks.

[0068] Robot task trajectory data can include video frame sequences, robot motion sequences, and task attribute information.

[0069] The video frame sequence is used to record visual observation information during the robot's task execution process. It can include multiple video frames arranged in the order of the robot's task execution time, and each video frame can correspond to the visual observation information at different time positions during the robot's task execution process.

[0070] For example, when a robot performs an object grasping task, the video frame sequence can include continuous visual observation information during the process of the robot recognizing the target object, moving the robotic arm, adjusting the attitude of the end effector, completing the grasping, and moving the target object.

[0071] In one implementation, each video frame in the video frame sequence corresponds to a video timestamp, which is used to indicate the time position of the video frame during the execution of the robot task.

[0072] Robot motion sequences are used to record the motion state information of a robot during task execution. A robot motion sequence can include multiple motion state information items arranged according to task execution time, and each motion state information item can correspond to the robot's motion state at different execution times.

[0073] For example, a robot action sequence may include robot joint state information, end effector state information, robotic arm motion state information, robot pose change information, and other information that can represent the robot's execution behavior.

[0074] In one implementation, each action state information in the robot action sequence corresponds to an action timestamp, which is used to indicate the execution time of the corresponding action state.

[0075] Furthermore, based on the correspondence between video timestamps and robot motion sequences, the video frames in the video frame sequence can be associated with the motion states in the robot motion sequence, so that visual observation information and motion state information correspond to the same task execution process.

[0076] Task attribute information is used to record task-related information corresponding to the tasks performed by the robot.

[0077] In one implementation, the task attribute information may include at least one of robot state machine stage information, task tags, skill identifiers, and language instruction information.

[0078] Among them, the robot state machine stage information can be used to represent the task stage that the robot is in during the execution of a task, such as the task preparation stage, the execution stage, or the completion stage.

[0079] Task tags can be used to indicate the task category corresponding to the task the robot is currently performing. For example, task tags can be used to distinguish between grasping tasks, assembly tasks, manipulation tasks, or other robot task types.

[0080] Skill identifiers can be used to represent the type of skill a robot uses to perform a task. For example, skill identifiers can represent grasping skills, localization skills, movement skills, or manipulation skills.

[0081] Language instruction information can be used to represent natural language descriptions of robot tasks. For example, language instruction information may include user-inputted descriptions of task objectives or task execution instructions.

[0082] In one implementation, robot task trajectory data can be acquired in real time by a robot data acquisition system during the robot's task execution. The robot data acquisition system can acquire video data collected by the robot's vision acquisition device, motion status data recorded by the robot's control system, and task attribute information provided by the task management system, and form robot task trajectory data based on the temporal correlation between different data sets.

[0083] In another implementation, robot mission trajectory data can also be obtained from historical robot mission datasets. The terminal device or server can read multiple pre-stored robot mission trajectory data sets and perform subsequent processing on the acquired robot mission trajectory data.

[0084] It should be noted that robot task trajectory data is not limited to a single fixed data organization format; it can be stored in files, database records, or other forms. Any data that can be linked to video frame sequences, robot motion sequences, and task attribute information during the robot's task execution can be considered robot task trajectory data.

[0085] The above methods can be used to obtain robot task trajectory data containing video information, motion information, and task-related information during the robot task execution process, providing a data foundation for the subsequent generation of video latent variables based on the robot task trajectory data.

[0086] Step S202: Determine the task timing sensitivity level of the robot task trajectory data based on the task attribute information.

[0087] In step S202, the corresponding task timing sensitivity level can be determined based on the task attribute information in the robot task trajectory data.

[0088] Task timing sensitivity level is used to characterize the sensitivity of a robot's task operations to changes in the temporal sequence during task execution. Different robot tasks have different requirements for maintaining the sequence of actions and temporal relationships due to differences in task objectives, execution stages, and operational requirements.

[0089] In one implementation, the task timing sensitivity level can be determined based on task attribute information. For example, when the robot state machine stage information indicates that the task is in a critical execution stage such as precise alignment, insertion, contact establishment, or state switching, the task process has high requirements for the sequence of actions and temporal correlation, and a high task timing sensitivity level can be determined. When the task label or skill identifier indicates that the task is an operation such as grasping, placing, or general handling, the task process has a certain tolerance for changes in temporal sequence, and a medium task timing sensitivity level can be determined. When the task label or skill identifier indicates that the task is a process such as navigation, waiting, or static observation, the sensitivity to changes in temporal sequence is low, and a low task timing sensitivity level can be determined.

[0090] In one implementation, the timing sensitivity level of a task can be determined by preset rules. Specifically, a set of task timing sensitivity levels can be preset, which may include high timing sensitivity level, medium timing sensitivity level, and low timing sensitivity level.

[0091] Among them, a high time sensitivity level can indicate the type of task that requires a strict time sequence relationship during the execution of the robot task; a medium time sensitivity level can indicate the type of task that has time correlation requirements in some operation stages during the execution of the robot task; and a low time sensitivity level can indicate the type of task that has a high tolerance for changes in time sequence during the execution of the robot task.

[0092] Furthermore, a first correspondence table can be obtained, which is used to represent the correspondence between the set of task time sensitivity levels and at least one of robot state machine stage information, task labels, and skill identifiers.

[0093] In one implementation, when the task attribute information of the robot task trajectory data includes at least one of robot state machine stage information, task tag, or skill identifier, the task timing sensitivity level corresponding to the robot task trajectory data is determined according to the first correspondence table of robot task trajectory data.

[0094] For example, when the robot state machine stage information represents a critical execution stage such as precise alignment, insertion, contact establishment, or state switching, it indicates that the sequence of actions and the temporal relationship during task execution are relatively important, and a high task temporal sensitivity level is determined based on the first correspondence table of robot task trajectory data; when the task label or skill identifier represents an operation such as grasping, placing, or general handling, a medium task temporal sensitivity level is determined based on the first correspondence table of robot task trajectory data; when the task label or skill identifier represents a process such as navigation, waiting, or static observation, a low task temporal sensitivity level is determined based on the first correspondence table of robot task trajectory data.

[0095] In another implementation, when the task attribute information does not include robot state machine stage information, task tags, and skill identifiers, task characteristics can be determined based on language instruction information, and the task timing sensitivity level can be determined based on the identified task characteristics.

[0096] Specifically, semantic analysis can be performed on language instruction information to obtain the corresponding task category or task characteristics.

[0097] For example, when the language instruction information contains task descriptions such as precise insertion, alignment, or button pressing, the corresponding high time-sensitive task features can be identified. When the language instruction information contains task descriptions such as moving to a specified location or transporting an object, the corresponding low time-sensitive task features can be identified.

[0098] Using the above method, the time sensitivity level of a task can be determined based on the task attribute information corresponding to the robot's task trajectory data. This allows the time processing requirements of the robot task to be represented in a hierarchical form, providing a task-level basis for determining the video processing method based on task characteristics.

[0099] Step S203: Determine the motion complexity level of the robot task trajectory data based on the video frame sequence and the robot motion sequence.

[0100] In step S203, the motion complexity level corresponding to the robot task trajectory data can be determined based on the video frame sequence and robot motion sequence in the robot task trajectory data.

[0101] In some embodiments of this application, the motion complexity level is used to characterize the degree of change in video content and robot movements during robot task execution. Therefore, the corresponding motion complexity level can be determined based on the change information in the video frame sequence and the robot movement sequence. Different robot tasks, due to differences in task objectives, execution processes, and operational requirements, result in variations in video and movement changes, thus leading to different motion complexity levels.

[0102] For example, when a robot performs tasks such as rapid movement, target tracking, or object handling, the target position in the video changes significantly, and the robot's joint state, end effector state, or robot pose also changes noticeably, corresponding to high motion complexity. When a robot performs tasks such as waiting, static observation, or maintaining a state, the video content and robot movements change less, corresponding to lower motion complexity.

[0103] In one implementation, reference Figure 3 , Figure 3 This is an exploded flowchart of an embodiment of step S203 in the video latent variable generation method according to this application. Step S203 may include the following steps S2031-S2034.

[0104] Step S2031: Generate video motion features of the video frame sequence based on the video content change information between adjacent video frames in the video frame sequence.

[0105] Specifically, the change information between the video content corresponding to each adjacent video frame in the video frame sequence can be analyzed. Based on the analysis results, the change information of the video content corresponding to each adjacent video frame can be obtained, and then the video motion features of the video frame sequence can be generated based on the change information of the video content corresponding to each adjacent video frame.

[0106] In one implementation, the video content change information between adjacent video frames can be represented by the optical flow information between adjacent video frames.

[0107] Furthermore, the video motion features corresponding to each adjacent video frame can be determined based on the optical flow information between adjacent video frames.

[0108] For example, optical flow information corresponding to each adjacent video frame can be calculated based on the pixel motion between adjacent video frames, and video motion features corresponding to the video frame sequence can be generated based on the optical flow information of each adjacent video frame.

[0109] Among them, video motion features can be used to characterize the overall motion level of the video corresponding to the video frame sequence, the degree of video motion change, and the peak of video motion.

[0110] For example, video motion features may include information such as the average optical flow, standard deviation of optical flow, and P99 of optical flow, which characterize a sequence of video frames.

[0111] Among them, the average optical flow is used to describe the overall visual motion level of the video frame sequence; the standard deviation of optical flow is used to describe the degree of fluctuation of visual motion in the video frame sequence over time; and the optical flow P99 is used to describe the peak of violent motion that occurs at a small number of time points.

[0112] Furthermore, optical flow P99 can be used to supplement the insufficient representation of local high-speed motion information by average optical flow. For example, a video frame sequence may be stationary for most of the time, but when a gripper closes or a robotic arm makes a rapid turn, a short period of high-speed motion occurs. Using only average optical flow may lead to the video frame sequence being judged as a low-motion state, while optical flow P99 can retain the ability to identify this high-speed motion phase, thereby improving the accuracy of representing sudden motion states during robot task execution.

[0113] Step S2032: Generate motion features of the robot action sequence based on the motion change information between adjacent robot actions in the robot action sequence.

[0114] Specifically, the motion change information between adjacent robot actions in the robot action sequence can be analyzed. Based on the analysis results, the motion change information corresponding to each adjacent robot action can be obtained, and then the motion motion features of the robot action sequence can be generated based on the motion change information corresponding to each adjacent robot action.

[0115] In one embodiment, the motion change information between adjacent robot actions can be represented by motion change information between at least one of the following: the motion of adjacent robot joints, the motion of the end effector, or other information that can characterize robot motion.

[0116] Furthermore, the motion characteristics corresponding to each adjacent robot action can be determined based on the motion change information between each adjacent robot joint motion action, end effector motion action, or other information that can characterize robot motion action, and the motion characteristics of the robot motion sequence can be determined based on the motion characteristics corresponding to each adjacent robot action.

[0117] In one alternative implementation, each robot action in the robot action sequence can be represented by a robot action vector.

[0118] In one alternative implementation, the motion change information between adjacent actions can be determined according to the following formula. .

[0119] in, Indicates the first action in the robot's action sequence The robot motion vector corresponding to each motion index. Indicates the first action in the robot's action sequence The robot motion vector corresponding to each motion index. This represents the motion change information between adjacent robot actions. The motion change information can be calculated based on the motion change rate, which is obtained from the time interval between adjacent robot actions. Indicates the first The robot motion vector corresponding to the i-th motion index is related to the i-th motion index. The time interval between robot motion vectors corresponding to each motion index.

[0120] The action index is used to represent the position number of the robot action in the robot action sequence.

[0121] Furthermore, motion characteristics of robot motion sequences can be generated based on motion change information between adjacent robot actions.

[0122] Among them, motion features can be used to characterize the overall level of motion change, degree of motion change, and peak value of motion change corresponding to robot motion sequences.

[0123] For example, motion characteristics may include information such as the average rate of motion change, the standard deviation of the rate of motion change, and the P99 of the rate of motion change, which characterize the robot's motion sequence.

[0124] Among them, the average motion change rate is used to describe the overall speed of the robot's motion sequence; the standard deviation of the motion change rate is used to describe the degree of change of the robot's motion speed over time in the robot's motion sequence; and the motion change rate P99 is used to describe the intense controlled motion that occurs in a short period of time.

[0125] Furthermore, the motion change rate P99 can be used to address the issue of insufficient representation of local high-speed motion states by the standard deviation of motion change rate. For example, when a robot performs high-speed uniform motion, although the motion change rate is high, the standard deviation of motion change rate may still be low due to the small speed change. Using only the standard deviation of motion change rate may not accurately reflect this motion state, while the motion change rate P99 can retain the ability to identify short-term high-speed controlled motion, thereby improving the accuracy of representing the overall motion level, degree of change, and sudden motion peaks in the robot's motion sequence.

[0126] In one implementation, when the information characterizing the robot's motion is the increment of motion of adjacent robot joints or the increment of motion of adjacent end effectors, the motion change information between adjacent robot actions can be obtained by dividing the increment of motion of adjacent robot joints or the increment of motion of end effectors by the time interval between adjacent robot actions.

[0127] In one implementation, when the information characterizing the robot's motion is the speed of adjacent robot joints or the speed of adjacent end effectors, the speed of adjacent robot joints or the speed of adjacent end effectors can be determined as motion change information between adjacent robot actions.

[0128] When dealing with robot motion information that includes multiple dimensions such as translation, rotation, joint movement, and gripper state, the physical units and numerical ranges corresponding to different dimensions may differ. Therefore, when calculating motion change information, the information representing robot motion can be normalized according to the physical range, data standard deviation, or preset scale corresponding to each dimension. Then, motion change information can be calculated based on the normalized information to avoid the impact of differences in physical scales on motion change information.

[0129] Step S2033: Determine the motion complexity corresponding to the robot task trajectory data based on the video motion features and action motion features.

[0130] In step S2033, the motion features of the video and the motion features of the action can be comprehensively analyzed to obtain the motion complexity corresponding to the robot task trajectory data.

[0131] In one implementation, video motion features and action motion features can be fused and calculated to obtain the corresponding motion complexity.

[0132] Specifically, motion complexity It can be calculated using the following formula: in, , and These represent the mean optical flow, the standard deviation of optical flow, and the optical flow P99, respectively. , and These represent the average rate of change of motion, the standard deviation of the rate of change of motion, and the rate of change of motion P99, respectively. , , , , and The weights are non-negative and can satisfy the condition that the sum of all weights is 1.

[0133] Using the above methods, the degree of motion change during robot task execution can be evaluated from both the dimension of video content change and the dimension of robot motion change.

[0134] Step S2034: Obtain the preset motion complexity level classification threshold, and determine the motion complexity level of the robot task trajectory data according to the motion complexity level classification threshold.

[0135] Specifically, a threshold for classifying motion complexity levels can be preset, and the motion complexity can be determined accordingly. The relationship between motion complexity level classification threshold and motion complexity level classification threshold determines motion complexity. The corresponding motion complexity level.

[0136] For example, motion complexity levels can be divided into high motion complexity level, medium motion complexity level, and low motion complexity level based on the motion complexity level classification threshold.

[0137] Specifically, the motion complexity can be obtained by referring to the following formula. Motion complexity level : in, and This represents the threshold for classifying motion complexity levels, and Specifically, the threshold for classifying motion complexity levels can be determined based on the statistical distribution of motion complexity of each robot task trajectory data in the existing training dataset. For example, it can be set according to the motion complexity quantile, or determined through pre-calibration experiments.

[0138] Among them, high motion complexity level can indicate that there are large-scale or rapid changes in video content and robot movements during the execution of robot tasks; medium motion complexity level can indicate that there are some changes in video content and robot movements during the execution of robot tasks; low motion complexity level can indicate that there are relatively small changes in video content and robot movements during the execution of robot tasks.

[0139] Using the above method, the motion complexity level corresponding to the robot task trajectory data can be determined based on the video frame sequence and the robot motion sequence, so that the degree of motion change during the robot task execution process can be represented in the form of a level.

[0140] Step S204: Determine the video processing configuration based on the task's timing sensitivity level and motion complexity level.

[0141] In step S204, the video processing configuration can be determined based on the task timing sensitivity level and motion complexity level.

[0142] The video processing configuration represents the video processing parameters determined for the current robot task trajectory data. The video processing configuration includes target motion configuration parameters and the target temporal downsampling factor corresponding to the video variational autoencoder.

[0143] The action configuration parameters are used to determine the minimum number of video frames required for a single training sample, which is also the minimum number of subsequent target temporal segment intervals. Action configuration parameters can include the number of historical observation steps. Predicted steps and the observation sampling span .

[0144] Time-series downsampling factor Used to represent the compression ratio of a video variational autoencoder in the time dimension, and to control the temporal coding granularity in the video latent variable generation process.

[0145] In one implementation, reference Figure 4 , Figure 4 This is an exploded flowchart of an embodiment of step S204 in the video latent variable generation method according to this application. Step S204 may include the following steps S2041-S2042.

[0146] Step S2041: Obtain the preset set of candidate action configuration parameters and the set of candidate temporal downsampling factors.

[0147] Specifically, a set of candidate action configuration parameters and a set of candidate time-series downsampling factors can be pre-set.

[0148] The candidate temporal downsampling factor set includes at least one candidate temporal downsampling factor, and different candidate temporal downsampling factors can correspond to different temporal coding granularities.

[0149] The candidate action configuration parameter set includes at least one candidate action configuration parameter, and different candidate action configuration parameters can correspond to different minimum video frame numbers. .

[0150] Candidate action configuration parameter set It can be represented as: in, This indicates the first in the set of candidate action configuration parameters. There are 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ...

[0151] No. The specific configuration parameters for each candidate action can be represented as follows: in, Indicates the first The number of historical observation steps corresponding to each candidate action configuration parameter. Indicates the first The number of action prediction steps corresponding to each candidate action configuration parameter. Indicates the first The observation sampling span corresponding to the configuration parameters of each candidate action.

[0152] Multiple candidate action configuration parameters can be included, for example: (2,4,1), (4,8,1), (8,16,2).

[0153] In some alternative implementations, the result can be obtained according to the following formula at the first... Under the given candidate action configuration parameters, the minimum number of video frames required for a single target temporal segmentation interval. : Step S2042: Based on the task timing sensitivity level and motion complexity level, determine the target timing downsampling factor from the candidate timing downsampling factor set, and determine the target action configuration parameter from the candidate action configuration parameter set.

[0154] Here, after obtaining the task timing sensitivity level, motion complexity level, candidate action configuration parameter set, and candidate timing downsampling factor set, the target timing downsampling factor can be determined from the candidate timing downsampling factor set based on the task timing sensitivity level and motion complexity level, and the target action configuration parameter can be determined from the candidate action configuration parameter set.

[0155] In one optional implementation, the target temporal downsampling factor can be determined from the set of candidate temporal downsampling factors according to a preset second correspondence table used to characterize the correspondence between the task temporal sensitivity level, motion complexity level and each candidate temporal downsampling factor.

[0156] For example, a second correspondence table can be pre-established between different task time sensitivity levels and different motion complexity levels and each candidate time downsampling factor.

[0157] After obtaining the task's temporal sensitivity level and motion complexity level, the second correspondence table mentioned above can be queried to select one candidate temporal downsampling factor from among the candidate temporal downsampling factors, and this candidate temporal downsampling factor can be determined as the target temporal downsampling factor.

[0158] For example, the candidate time-series downsampling factor set may include three candidate time-series downsampling factors: 2, 4, and 8.

[0159] For different task time sensitivity levels and motion complexity levels, the target time downsampling factor can be determined using the second correspondence table shown in Table 1.

[0160] Table 1: Second Correspondence Table According to the second correspondence table above, when the robot task trajectory data corresponds to a high temporal sensitivity level, a smaller target temporal downsampling factor of 2 is selected regardless of the motion complexity level to retain higher temporal dimension information. When the robot task trajectory data corresponds to a medium temporal sensitivity level, the target temporal downsampling factor can be selected based on the motion complexity level, with 4 corresponding to medium and low motion complexity, and 2 corresponding to high motion complexity. When the robot task trajectory data corresponds to a low temporal sensitivity level, a larger target temporal downsampling factor can be selected based on the motion complexity level to reduce the computational overhead in the video encoding process, with 8 corresponding to low motion complexity, 4 corresponding to medium motion complexity, and 2 corresponding to high motion complexity.

[0161] The above method can adaptively determine the target temporal downsampling factor of the video variational autoencoder based on the task's temporal sensitivity and motion complexity, so that tasks with high temporal sensitivity or high motion complexity retain more temporal dimension information, while reducing the video encoding processing overhead for tasks with low temporal sensitivity or low motion complexity.

[0162] Furthermore, the target action configuration parameters can be determined from the set of candidate action configuration parameters based on a pre-defined third correspondence table that characterizes the relationship between the task's temporal sensitivity level, motion complexity level, and each candidate action configuration parameter.

[0163] For example, a third correspondence table can be pre-established between different task timing sensitivity levels and different motion complexity levels and the configuration parameters of each candidate action.

[0164] After obtaining the task's timing sensitivity level and motion complexity level, you can query the third correspondence table mentioned above, select one of the candidate action configuration parameters from the candidate action configuration parameters, and determine that candidate action configuration parameter as the target action configuration parameter.

[0165] For example, for robot tasks with high temporal sensitivity and high motion complexity, since the actions change rapidly during task execution and the correlation between different time states is strong, in order to retain more temporal dimension information, a smaller candidate temporal downsampling factor can be selected as the target temporal downsampling factor, and a candidate action configuration parameter with a smaller minimum number of video frames can be selected as the target action configuration parameter, so as to improve the ability to represent the key action processes of the task.

[0166] For robot tasks with low temporal sensitivity and low motion complexity, since the state changes during task execution are relatively slow, a larger candidate temporal downsampling factor can be selected as the target temporal downsampling factor to reduce the computational overhead in the video variational autoencoder encoding process. At the same time, candidate action configuration parameters with a larger minimum number of video frames can be selected as target action configuration parameters to reduce the number of video segments and improve the overall data processing efficiency.

[0167] For robot tasks with low temporal sensitivity and low motion complexity, the largest candidate temporal downsampling factor can be selected as the target temporal downsampling factor to reduce the computational cost of video encoding, and at the same time, the candidate action configuration parameters corresponding to the larger minimum number of video frames can be selected as the target action configuration parameters.

[0168] In another implementation, the target action configuration parameters can also be determined by configuring the selection model.

[0169] Specifically, the task timing sensitivity level and / or motion complexity level can be input into the preset configuration selection model to obtain the selection probability corresponding to each candidate action configuration parameter; based on the selection probability corresponding to each candidate action configuration parameter, the candidate action configuration parameter with the highest selection probability is determined; and the candidate action configuration parameter with the highest selection probability is determined as the target action configuration parameter.

[0170] The configuration selection model can employ a learned selection model. This model evaluates candidate action configuration parameters based on the task temporal sensitivity level and motion complexity level corresponding to the current robot task trajectory data, and selects a complete set of candidate action configuration parameters as the target action configuration parameters.

[0171] Specifically, we can first obtain the feature vectors of the input learning selection model. eigenvectors It may include at least one of the following: motion complexity level, task timing sensitivity level, or video motion features, action motion features, and task attribute information.

[0172] Then, the feature vector The configuration parameters of each candidate action are input into the learning-based selection model, and the learning-based selection model selects actions based on feature vectors. Output the selection score corresponding to each candidate action configuration parameter, where the selection score set corresponding to each candidate action configuration parameter is... Specifically, this can be expressed by the following formula: in, Indicates that it has trainable parameters The learning-based selection model, This represents the set of selection scores corresponding to the configuration parameters of each candidate action, and .

[0173] in, Indicates the first The unnormalized selection score corresponding to each candidate action configuration parameter. This indicates the number of candidate action configuration parameters.

[0174] It should be noted that a higher score indicates a higher degree of compatibility between the corresponding candidate action configuration parameters and the current robot task trajectory data, while a lower score indicates a lower degree of compatibility between the corresponding candidate action configuration parameters and the current robot task trajectory data.

[0175] During the training phase, temperature parameters can be used to optimize the selection of configuration parameters for each candidate action. Convert the unnormalized selection scores corresponding to the configuration parameters of each candidate action into selection probabilities.

[0176] Among them, the The selection probability corresponding to each candidate action configuration parameter This can be expressed by the following formula: in, In the eigenvector Under the condition of choosing the first The probability of selecting each candidate action configuration parameter; This represents the summation index, used for sequential traversal. Configuration parameters for each candidate action; Indicates the first The selection score corresponding to the configuration parameters of each candidate action; Represents an exponential function; This indicates a temperature parameter greater than 0.

[0177] The probabilities of each choice satisfy: and: Temperature parameters Used to adjust the smoothness of the selection probability distribution. When the temperature parameter... When the temperature parameter is relatively small, the selection probability is more concentrated on the candidate action configuration parameters with higher scores; when the temperature parameter is relatively small... When the value is large, the selection probability corresponding to each candidate action configuration parameter is smoother. Temperature parameter Discrete selection optimization used during the training phase does not change the predefined parameter values ​​in the configuration parameters of each candidate action.

[0178] In one implementation, the candidate action configuration parameter with the highest score can be selected as the target action configuration parameter.

[0179] The target action configuration parameter number can be represented as ,in, It can be obtained through the following formula: in, This indicates the candidate action configuration parameter number with the highest selection score, which is also the target action configuration parameter number. This indicates that the candidate action configuration parameter number that makes the selected score the maximum value is returned.

[0180] because The function maintains a monotonic relationship with the selection score. The candidate action configuration parameter corresponding to the highest selection score is the candidate action configuration parameter corresponding to the highest selection probability. Therefore, the candidate action configuration parameter with the highest selection probability can also be used as the target action configuration parameter.

[0181] Target action configuration parameter number It can also be obtained using the following formula: in, This indicates the candidate action configuration parameter number with the highest selection probability, which is also the target action configuration parameter number. This indicates the return of the candidate action configuration parameter number that maximizes the selection probability.

[0182] For example, when the candidate action configuration parameters include (2,4,1), (4,8,1), and (8,16,2), the corresponding candidate action configuration parameter numbers are 1, 2, and 3, respectively.

[0183] The corresponding selection probabilities can be 0.15, 0.70, and 0.15, respectively. Since the highest selection probability is 0.7, the candidate action configuration parameter number 2 corresponding to this selection probability can be determined as the target action configuration parameter number. The value is 2.

[0184] In one implementation, when multiple candidate action configuration parameters have the same maximum selection score or selection probability, the target action configuration parameter can be determined from the multiple candidate action configuration parameters with the same selection score or selection probability according to the preset priority of candidate action configuration parameters, the calculation cost of candidate action configuration parameters, the configuration switching range of candidate action configuration parameters, or the candidate number order.

[0185] Determine the target action configuration parameter number Then, the number can be... The corresponding candidate action configuration parameters are used as the target action configuration parameters. It can be represented as: in, It can represent the target action configuration parameters. The number of historical observation steps used Indicates the target action configuration parameters The number of steps predicted by the action is used. Indicates the target action configuration parameters The observation sampling span used.

[0186] For example, it can be The configuration parameters for the candidate action corresponding to 2 are determined as the configuration parameters for the target action, that is, It can be (4,8,1).

[0187] Here, since the selection of candidate action configuration parameters, construction of video frame indexes, and generation of temporal segment intervals all involve discrete operations, the learning-based selection model can be trained using policy gradients, discrete routes with pass-through estimation, or expected loss based on multiple candidate branches.

[0188] Total loss corresponding to the learning-based selection model It can be represented as: in, Indicates the loss from action prediction; This indicates the degree of inconsistency between predicted action changes and actual action changes near the boundaries of time-series segment intervals; This indicates the input length, video memory usage, or inference latency cost corresponding to the current target action configuration parameters; This represents a routing constraint used to avoid selecting the same candidate action configuration parameter from a large dataset of robot task trajectories. This represents the switching cost used to suppress frequent switching of action configuration parameters within the same robot task trajectory data; , , and These represent the weights of the corresponding loss terms.

[0189] Using the above method, the video processing configuration can be adaptively determined based on the task's temporal sensitivity level and motion complexity level corresponding to the robot task.

[0190] Step S205: Based on the video processing configuration, determine multiple target temporal segmentation intervals in the video frame sequence that satisfy the temporal coding constraints of the video variational autoencoder.

[0191] In step S205, multiple target temporal segmentation intervals can be determined in the video frame sequence corresponding to the robot task trajectory data according to the video processing configuration determined in step S204.

[0192] The target temporal segment interval is used as the input video segment for subsequent video variational autoencoder encoding. Different target temporal segment intervals correspond to different video frame index ranges in the video frame sequence.

[0193] Here, the video frame index can refer to the position number of the video frame in the video frame sequence.

[0194] Since video variational autoencoders typically need to meet specific temporal coding requirements, the number of video frames in the target temporal segment interval must match the temporal coding scheme of the video variational autoencoder. Therefore, when determining the target temporal segment interval, each target temporal segment interval needs to satisfy the temporal coding constraints of the video variational autoencoder.

[0195] Among them, the temporal coding constraints of the video variational autoencoder include the temporal alignment constraints between the number of input video frames and the target temporal downsampling factor.

[0196] For example, when the target temporal downsampling factor corresponding to the video variational autoencoder is 4, the number of input video frames needs to satisfy the time alignment constraint relationship with the target temporal downsampling factor, that is, the number of input video frames should satisfy the condition that it is divisible by 4. Specifically, when the number of input video frames is 16, after compression of the time dimension corresponding to the target temporal downsampling factor 4, the corresponding 4 temporal dimension latent variables can be mapped, thereby ensuring that after encoding by the video variational autoencoder, the video frame index range corresponding to the target temporal segment interval can be effectively mapped to the latent variable time index range.

[0197] When the number of input video frames is 15, the above time alignment constraint cannot be met. Therefore, the number of input video frames can be adjusted to meet the time coding requirements of the video variational autoencoder.

[0198] In one implementation, reference Figure 5 , Figure 5 This is an exploded flowchart of an embodiment of step S205 in the video latent variable generation method according to this application. Step S205 may include the following steps S2051-S2056.

[0199] Step S2051: Determine the temporal segmentation interval length of the video frame sequence based on the target action configuration parameters and the target temporal downsampling factor.

[0200] The temporal segmentation interval length represents the number of video frames required for each target temporal segmentation interval, determined by segmenting the video frame sequence according to the target motion configuration parameters and the target temporal downsampling factor. In other words, it represents the number of video frames required for each target temporal segmentation interval to satisfy the temporal coding constraints corresponding to the video variational autoencoder.

[0201] It should be noted that when the minimum number of video frames required for the target temporal segmentation interval satisfies the temporal coding constraints of the video variational autoencoder, this minimum number of video frames can be the length of the temporal segmentation interval.

[0202] When the minimum number of video frames required for the target temporal segmentation interval does not meet the temporal coding constraints of the video variational autoencoder, the minimum number of video frames for the target temporal segmentation interval can be adjusted upward in the time dimension according to the target temporal downsampling factor to obtain the temporal segmentation interval length that meets the temporal coding constraints of the video variational autoencoder.

[0203] Specifically, when the minimum number of video frames required for the target temporal segmentation interval does not meet the temporal coding constraints of the video variational autoencoder, the length of the temporal segmentation interval can be obtained using the following formula. : in, Indicates the length of the time series segment interval. This represents the minimum number of video frames required for the target temporal segmentation interval. Indicates the target time series downsampling factor. This indicates the rounding up operation.

[0204] Furthermore, the length of the time series segment interval The following constraints must be met: This indicates that the length of the temporal segment interval is divisible by the target temporal downsampling factor, thus satisfying the temporal coding constraints of the video variational autoencoder.

[0205] at the same time: This indicates that the length of the temporal segment interval that satisfies the temporal coding constraints of the video variational autoencoder is not less than the minimum number of video frames required for the target temporal segment interval.

[0206] Furthermore: This indicates that the number of video frames added during the temporal coding constraint alignment process of the video variational autoencoder is less than the number of video frames corresponding to a target temporal downsampling factor, thereby reducing video frame redundancy caused by temporal alignment.

[0207] By using the above method, while ensuring the coverage requirements of robot task action information, the number of video frames required for a single target temporal segmentation interval can meet the temporal coding constraints of the video variational autoencoder.

[0208] Step S2052: Divide the video frame sequence according to the length of the time segment interval to obtain multiple initial time segment intervals.

[0209] Specifically, starting from the index of the first video frame in the video frame sequence according to the length of the time-series segmentation interval, the video frame sequence can be continuously divided according to the time order of the video frames to obtain multiple initial time-series segmentation intervals.

[0210] For example, when a video frame sequence contains multiple consecutive video frames, the corresponding number of video frames can be extracted sequentially according to the length of the temporal segment interval to form multiple consecutive initial temporal segment intervals.

[0211] Among them, for the first Initial time series segmentation interval It can be expressed by the following formula: in, Indicates the first An initial time series segmentation interval Indicates the first The starting video frame index of each initial time-series segment interval. Indicates the first The index of the end video frame of each initial time-series segment interval.

[0212] Step S2053: Determine whether there are any initial temporal segmentation intervals among the multiple initial temporal segmentation intervals that do not satisfy the temporal coding constraints of the video variational autoencoder.

[0213] After obtaining multiple initial temporal segment intervals, it can be determined whether the number of video frames corresponding to each initial temporal segment interval meets the temporal coding constraints of the video variational autoencoder.

[0214] For example, the following formula can be used to determine whether the number of video frames corresponding to each initial temporal segment interval satisfies the temporal coding constraints of the video variational autoencoder: in, express and The number of video frames is relatively small; Indicates the length of the time series segment interval; This indicates the total number of video frames in the video frame sequence. Indicates the first The starting video frame index corresponding to each initial time-series segment interval.

[0215] when: This indicates that the number of video frames corresponding to the initial temporal segmentation interval satisfies the temporal coding constraints of the video variational autoencoder.

[0216] when: This indicates that the number of video frames corresponding to the initial temporal segmentation interval does not meet the temporal coding constraints of the video variational autoencoder.

[0217] Step S2054: If not, determine the multiple initial temporal segmentation intervals as multiple target temporal segmentation intervals that satisfy the temporal coding constraints of the video variational autoencoder.

[0218] If there is no initial temporal segmentation interval among the multiple initial temporal segmentation intervals that does not satisfy the temporal coding constraints of the video variational autoencoder, it means that the number of video frames corresponding to the multiple initial temporal segmentation intervals all meet the temporal segmentation interval length.

[0219] At this point, multiple initial time series segment intervals can be directly determined as multiple target time series segment intervals.

[0220] Step S2055: If it exists, adjust the initial temporal segmentation interval that does not meet the temporal coding constraints of the video variational autoencoder according to the temporal segmentation interval length to obtain the adjusted initial temporal segmentation interval.

[0221] The adjusted initial temporal segmentation interval satisfies the temporal coding constraints of the video variational autoencoder.

[0222] In other words, the number of video frames in the adjusted initial temporal segment interval is equal to the length of the temporal segment interval.

[0223] It should be noted that among multiple initial temporal segment intervals, the last initial temporal segment interval may fail to meet the temporal segment interval length due to insufficient remaining video frames, that is, it does not meet the temporal coding constraints of the video variational autoencoder.

[0224] In some optional implementations, when there are initial temporal segmentation intervals among multiple initial temporal segmentation intervals that do not meet the temporal coding constraints of the video variational autoencoder, the number of supplementary video frames to be added can be determined based on the length of the temporal segmentation interval. Supplementary video frames are generated based on the number of supplementary video frames. The supplementary video frames are added to the initial temporal segmentation intervals that do not meet the temporal coding constraints of the video variational autoencoder to obtain the adjusted initial temporal segmentation intervals.

[0225] The adjusted initial time-series segmentation interval includes the original video frames and supplementary video frames.

[0226] Here, the original video frame can refer to the video frame contained in the initial temporal segmentation interval that does not satisfy the temporal coding constraints of the video variational autoencoder.

[0227] In one implementation, for an initial temporal segmentation interval that does not meet the temporal coding constraints of a video variational autoencoder, the number of original video frames corresponding to the original video frames contained therein can be recorded.

[0228] Furthermore, the number of supplementary video frames that need to be added can be determined based on the difference between the number of original video frames contained in the initial temporal segmentation region that does not meet the temporal coding constraints of the video variational autoencoder and the length of the temporal segmentation interval. Then, supplementary video frames can be generated based on the number of supplementary video frames.

[0229] In one implementation, supplementary video frames with a number of supplementary frames can be generated by boundary frame duplication, reflection filling, or preset filling, and the supplementary video frames can be added to the corresponding initial time segment interval.

[0230] In one implementation, supplementary video frames can be added to the left and / or right of the initial temporal segmentation interval to ensure that the number of video frames corresponding to the adjusted initial temporal segmentation interval meets the temporal coding constraints of the video variational autoencoder, that is, to meet the temporal segmentation interval length.

[0231] Furthermore, the effective frame mask can be used to distinguish between the original video frames and the supplementary video frames.

[0232] For example, the original video frame can be set as a valid frame identifier, while the supplementary video frame can be set as an invalid frame identifier.

[0233] By using an effective frame mask, the original video frames and supplementary video frames can be identified during the subsequent video variational autoencoder encoding process, thus avoiding the impact of supplementary video frames on the initial temporal segmentation intervals.

[0234] Step S2056: The initial temporal segmentation interval that satisfies the temporal coding constraints of the video variational autoencoder and the adjusted initial temporal segmentation interval are determined as multiple target temporal segmentation intervals that satisfy the temporal coding constraints of the video variational autoencoder.

[0235] Here, the initial temporal segmentation interval that satisfies the temporal coding constraints of the video variational autoencoder and the adjusted initial temporal segmentation interval are determined as multiple target temporal segmentation intervals. That is, multiple target temporal segmentation intervals all satisfy the temporal coding constraints of the video variational autoencoder.

[0236] Furthermore, each target time series segment interval It can be represented as: For any target time series segment interval, This can represent the starting video frame index corresponding to the target time-series segment interval. This represents the index of the last video frame corresponding to the target temporal segmentation interval, and the index of the last video frame... It is not included in the target time series segmentation interval.

[0237] The target temporal segmentation interval contains the following number of video frames: In one alternative implementation, after determining multiple target temporal segment intervals that satisfy the temporal coding constraints of the video variational autoencoder in the video frame sequence according to the video processing configuration, the target temporal segment intervals can be further optimized and adjusted by boundary.

[0238] In the above embodiments, when dividing the video frame sequence, the target temporal segment interval is mainly determined based on the length of the temporal segment interval, which may not fully consider the changes in actions during the execution of the robot task.

[0239] Therefore, the boundary position of the target time segment interval, that is, the starting video frame index and / or the starting video frame index position, may be located at a position where the robot accelerates, decelerates, changes its direction of motion, or switches its control state, resulting in the continuous robot motion process being divided into different target time segment intervals.

[0240] Based on this, while keeping the length of the target time segment interval, the target time downsampling factor, and the target motion configuration parameters unchanged, the boundary position of the target time segment interval, i.e. the starting video frame index and / or the starting video frame index position, can be optimized so that the optimized boundary position of the target time segment interval is located in a position where the robot motion change trend is relatively stable.

[0241] Specifically, firstly, for any target time series segment interval, within a preset boundary range, multiple candidate boundary adjustment positions are determined.

[0242] The candidate boundary adjustment position represents the boundary position to which the current boundary position of the target time series segment interval can be adjusted. The preset boundary range limits the adjustable range of the current boundary position of the target time series segment interval to avoid excessive boundary position adjustment.

[0243] In one implementation, the preset boundary range can be a video frame range formed by extending a preset number of video frames forward and / or backward along the video frame sequence, based on the current boundary position of the target time segment interval.

[0244] Here, the positions of multiple video frames within a preset boundary range can be determined as candidate boundary adjustment positions.

[0245] In addition, when a portion of the preset boundary range exceeds the effective index range of the video frame sequence, the preset boundary range can be truncated based on the starting and / or ending video frame indices of the video frame sequence, and only the video frame positions within the effective index range can be determined as candidate boundary adjustment positions.

[0246] The effective index range of the video frame sequence is used to represent the index range corresponding to the actual video frames contained in the video frame sequence.

[0247] By determining multiple candidate boundary adjustment positions near the current boundary position of the target time segment interval, we can limit the boundary adjustment range while providing a candidate range for selecting the target boundary adjustment position based on the changes in robot actions.

[0248] Furthermore, based on the video frame timestamps corresponding to the adjustment positions of each candidate boundary and the action timestamps corresponding to each robot action in the robot action sequence, the target action index corresponding to each candidate boundary adjustment position is determined.

[0249] Specifically, after obtaining multiple candidate boundary adjustment positions, the video frame index corresponding to each candidate boundary adjustment position can be determined, and the corresponding video frame timestamp can be obtained based on the video frame index.

[0250] It also determines the action index corresponding to each robot action in the robot action sequence, and obtains the corresponding action timestamp based on the action index.

[0251] It is important to understand that video frame sequences and robot motion sequences may have different sampling frequencies.

[0252] For example, video frame sequences can be generated according to the camera acquisition frequency, and robot motion sequences can be generated according to the robot control frequency.

[0253] Therefore, there is usually no direct one-to-one correspondence between video frame indexes and robot motion indexes, and the video frame index corresponding to the candidate boundary adjustment position cannot be directly used as the robot motion index.

[0254] The action index can refer to the position number of the robot action in the robot action sequence.

[0255] The candidate boundary needs to be adjusted and mapped from the video frame sequence to the robot motion sequence first.

[0256] Specifically, the target action index corresponding to each candidate boundary adjustment position is determined based on the video frame timestamp corresponding to each candidate boundary adjustment position and the action timestamp corresponding to each robot action in the robot action sequence.

[0257] For example, for any candidate boundary adjustment position, the candidate boundary adjustment position can be mapped from the video frame sequence to the robot motion sequence using the following formula. That is, the target motion index corresponding to each candidate boundary adjustment position can be determined using the following formula. : in, This indicates the video frame index corresponding to the candidate boundary adjustment position; This indicates that the position of each candidate action in the robot's action sequence should be adjusted for the candidate boundary. This represents the timestamp of the video frame corresponding to the video frame index y at the candidate boundary adjustment position. This represents the timestamp of the action corresponding to each candidate action index; This represents the absolute time difference between the timestamp of the action corresponding to the candidate action index and the timestamp of the video frame corresponding to the video frame index of the candidate boundary adjustment position. This represents the candidate action index that is closest to the candidate boundary adjustment position in terms of time difference, which is also the target action index.

[0258] In this way, the candidate boundary of the target temporal segment interval can be adjusted and mapped from the video frame sequence to the robot motion sequence.

[0259] Here, video frame timestamps and motion timestamps are used to time-align the video frame sequence and the robot motion sequence. The target motion index is used to indicate the arrangement position of the robot motion in the robot motion sequence that matches the video frame index corresponding to the candidate boundary adjustment position, so as to obtain the corresponding robot motion data according to the target motion index.

[0260] Here, when the camera acquisition frequency of the video frame sequence and the robot control frequency of the robot action sequence differ significantly, or when there is no sufficiently close corresponding moment between the video frame timestamp and the action timestamp, linear interpolation, pose interpolation, or other time synchronization methods can be used to obtain the target action index corresponding to the moment of the candidate boundary adjustment position.

[0261] Furthermore, based on the target action index corresponding to the position of each candidate boundary adjustment, the robot action data corresponding to each target action index is determined.

[0262] The robot motion data of the target motion index can refer to at least one of the following in the robot motion sequence at the location of the target motion index: translation data, rotation data, joint position, joint velocity, joint acceleration, end effector pose, gripper state, control commands, and force or torque information.

[0263] Furthermore, based on the motion data of each robot, the first boundary evaluation result corresponding to each candidate boundary adjustment position is determined.

[0264] Specifically, after determining the target action index corresponding to each candidate boundary adjustment position, robot action data for each target action index can be obtained.

[0265] Then, based on the robot motion data of the target motion index and the adjacent target motion indices, the second-order difference data of the robot motion for that target motion index is obtained.

[0266] The adjacent target action index can refer to the target action index that is adjacent to the target action index in the robot action sequence according to the action arrangement order.

[0267] In one implementation, the adjacent motion index used to calculate the second-order difference data of robot motions may include the previous target motion index and the previous two target motion indices.

[0268] For example, the target action index corresponding to a candidate boundary adjustment position is: Then the previous target action index corresponding to the target action index can be represented as: The first two target action indices can be represented as .

[0269] Accordingly, the target action index can be obtained. Action Index and action index The corresponding robot motion data.

[0270] It should be noted that robot motion data may include different motion dimensions such as translation data, rotation data, joint position, joint velocity, gripper control quantity, and torque control quantity. The physical units and numerical ranges corresponding to different motion dimensions may be different. Therefore, the robot motion data corresponding to each target motion index can be scaled and normalized to obtain normalized robot motion data.

[0271] For the normalized robot motion data, the target motion index The corresponding first-order difference data can be expressed by the following formula: in, Indicates the target action index The corresponding first-order difference data is used to characterize the target action index. The corresponding robot action index relative to the previous target action The corresponding range of change in the robot's movements; Indicates the target action index The corresponding robot motion data; Indicates the index of the previous action The corresponding robot motion data.

[0272] Furthermore, the index of the previous target action. The corresponding first-order difference data can be represented as: in, Indicates the index of the previous target action. The corresponding first-order difference data is used to characterize the index of the previous target action. The corresponding robot action relative to the index of the first two target actions The corresponding range of change in the robot's movements; Indicates the index of the first two target actions The corresponding robot motion data.

[0273] Target Action Index The corresponding second-order difference data of the robot action can be represented as: Substituting the two first-order difference data expressions, we can obtain: in, Indicates the target action index The corresponding second-order difference data of robot actions are used to characterize the degree of change between two adjacent first-order difference data.

[0274] Furthermore, it can be based on the target action index Based on the corresponding first-order difference data and second-order difference data of robot actions, the first boundary evaluation result corresponding to the candidate boundary adjustment position is determined: in, Indicates video frame index The first boundary evaluation result corresponding to the candidate boundary adjustment position; Indicates the target action index The corresponding first-order difference data of robot motion; Indicates the target action index The corresponding second-order difference data of robot motion; Represents the second norm of data; This represents the non-negative weights corresponding to the first-order difference data of the robot's actions; This represents the non-negative weights corresponding to the second-order difference data of the robot's actions.

[0275] The above method can be used to determine the first boundary evaluation result corresponding to the candidate boundary adjustment position.

[0276] The first boundary evaluation result is used to characterize the degree of change in robot action between the candidate boundary adjustment position and the nearby adjacent positions, and to evaluate whether the candidate boundary adjustment position is suitable as the boundary position after translation of the target time segment interval.

[0277] In some optional implementations, when the target action index is the starting action index of the robot action sequence, since the second-order difference data of the robot action of the starting action index cannot be obtained, the first boundary evaluation result of the candidate boundary adjustment position corresponding to the starting action index can also be determined by using the first-order difference data of the robot action of the starting action index or the one-sided difference method, or the corresponding candidate boundary adjustment position can be marked as an unselectable boundary position.

[0278] Next, the candidate boundary adjustment position with the smallest first boundary evaluation result is determined as the target boundary adjustment position.

[0279] The larger the first boundary evaluation result, the greater the degree of motion change between the candidate boundary adjustment position and the nearby adjacent action positions. The candidate boundary adjustment position may be in the process of robot acceleration, deceleration, change of motion direction or control state switching.

[0280] The smaller the first boundary evaluation result, the smaller the degree of motion change between the candidate boundary adjustment position and the nearby adjacent motion change amount, and the more stable the robot motion change trend at the candidate boundary adjustment position, thus making it more suitable as the boundary position after translation of the target time segment interval.

[0281] Specifically, after obtaining the first boundary evaluation results corresponding to each candidate boundary adjustment position, multiple first boundary evaluation results can be compared, and the candidate boundary adjustment position with the smallest first boundary evaluation result can be determined as the target boundary adjustment position.

[0282] In one implementation, when the first boundary evaluation results corresponding to multiple candidate boundary adjustment positions are the same, or the difference between multiple first boundary evaluation results is less than a preset boundary result threshold, the candidate boundary adjustment position that is closer to the current boundary position of the target time series segment interval can be selected as the target boundary adjustment position to reduce the boundary adjustment amplitude of the target time series segment interval.

[0283] Furthermore, the boundary translation amount is determined based on the positional difference between the target boundary adjustment position and the current boundary position of the target time-series segment interval.

[0284] Specifically, after determining the target boundary adjustment position, the boundary translation amount can be determined based on the positional difference between the video frame index corresponding to the target boundary adjustment position and the video frame index corresponding to the current boundary position of the target temporal segmentation interval.

[0285] Among them, the boundary translation amount is used to represent the translation direction and translation distance of the target temporal segment interval in the video frame sequence.

[0286] Specifically, by determining the boundary translation amount based on the position difference between the target boundary adjustment position and the current boundary position, the position change corresponding to the target boundary adjustment position can be converted into the overall translation parameter of the target time series segment interval.

[0287] Finally, based on the boundary translation amount, the starting and ending boundaries of the target time series segment interval are synchronously translated in the same direction and by the same distance to obtain the first target time series segment interval after translation.

[0288] The first target time series segment interval can refer to the target time series segment interval after being shifted according to the boundary shift amount.

[0289] Specifically, the start and end boundaries of the target temporal segment interval can be synchronously shifted according to a determined boundary shift amount. The number of video frames contained in the target temporal segment interval remains unchanged before and after the shift.

[0290] Since the number of video frames contained in the first target temporal segment interval after the translation remains unchanged, the first target temporal segment interval after the translation still satisfies the video variational autoencoder temporal coding constraint corresponding to the target temporal downsampling factor.

[0291] In one implementation, when the starting or ending video frame index of the first target temporal segment interval after synchronous translation may exceed the effective index range of the video frame sequence, the candidate boundary adjustment position can be excluded when determining the candidate boundary adjustment position, or the boundary translation amount can be limited according to the effective index range of the video frame sequence to ensure that the first target temporal segment interval after translation is within the effective index range of the video frame sequence.

[0292] Using the above method, multiple candidate boundary adjustment positions can be determined near the current boundary position of the target temporal segmentation interval. Then, based on the video frame timestamp and robot action timestamp, the target action index corresponding to each candidate boundary adjustment position is determined. Based on the target action index corresponding to each candidate boundary adjustment position, the robot action data corresponding to each target action index is determined. Based on the robot action data, the first boundary evaluation result corresponding to each candidate boundary adjustment position is determined, and the position with the smallest first boundary evaluation result is determined as the target boundary adjustment position. Finally, based on the position difference between the target boundary adjustment position and the current boundary position, the starting and ending video frame indices of the target temporal segmentation interval are synchronously translated. This ensures that, while maintaining the number of video frames and the temporal coding constraints of the video variational autoencoder, the boundary of the translated first target temporal segmentation interval is located at a position where the robot action change trend is relatively stable.

[0293] In one alternative implementation, when multiple target temporal segment intervals are used as independent training samples, after determining multiple target temporal segment intervals that satisfy the temporal coding constraints of the video variational autoencoder in the video frame sequence according to the video processing configuration, each target temporal segment interval can be subjected to overall translation optimization to obtain the corresponding second target temporal segment interval.

[0294] Here, mutually independent training samples can refer to the latent variables corresponding to multiple target time-series segment intervals being input into the model as independent training samples during the training process of the vision-language-action model, and the contextual information provided by each target time-series segment interval does not depend on the adjacent target time-series segment intervals.

[0295] When multiple target temporal segments are used as independent training samples, the specific temporal position of each segment within the video frame sequence can affect the video content, robot actions, and the matching relationship between the training sample and the action supervision target. For example, if the starting or ending video frame index of a target temporal segment is a location where robot actions change significantly, continuous robot action processes may be divided into different training samples, thus affecting the representation of complete action semantics by a single training sample.

[0296] Therefore, while keeping the number of video frames contained in the target temporal segmentation interval unchanged, the starting video frame index and the ending video frame index of the target temporal segmentation interval can be synchronously shifted, and the candidate temporal segmentation interval with the best boundary evaluation can be selected from multiple shifting results as the second target temporal segmentation interval.

[0297] Specifically, for any target time series segment interval, multiple legal candidate boundary translation amounts are determined.

[0298] Among them, the translation amount of the legitimate candidate boundary is used to represent the translation direction and translation distance of the target temporal segment interval along the time direction of the video frame sequence.

[0299] Here, the translation amount of each legal candidate boundary can be an integer multiple of the target temporal downsampling factor.

[0300] For example, let the translation amount of a valid candidate boundary be... The target time series downsampling factor is Then the legal candidate boundary translation can satisfy: Therefore, shifting the target temporal segment interval according to the shift amount of each legal candidate boundary is equivalent to shifting the target temporal segment interval along the video frame sequence by an integer number of video variational autoencoder latent variable time steps, thereby maintaining the phase correspondence between the boundary position of the shifted target temporal segment interval and the time grid of the video variational autoencoder.

[0301] Furthermore, the candidate temporal segmentation intervals formed based on the translation of each legitimate candidate boundary can be located within the effective video frame index range of the video frame sequence.

[0302] Since the aforementioned target time series segmentation interval can be represented as: The total number of indexes of the video frames contained in the video frame sequence is Then, the candidate time series segment intervals after being shifted by the legal candidate boundary shift amount can still satisfy the following formula: as well as in, This represents the starting video frame index corresponding to the candidate temporal segment interval formed after translation according to the legal candidate boundary translation amount; Indicates the index of the end video frame corresponding to the candidate time-series segment interval; This represents the total number of indexes of the video frames in the current video frame sequence.

[0303] By applying the above constraints, it can be ensured that the starting video frame index of the candidate time-series segment interval is no earlier than the starting video frame index of the video frame sequence, and the ending video frame index of the candidate time-series segment interval is no later than the ending video frame index of the video frame sequence, thereby ensuring that the candidate time-series segment interval is within the effective index range of the video frame sequence.

[0304] In one implementation, the legitimate candidate boundary translation can also satisfy constraints such as the completeness of historical observation data and action supervision targets, data validity, and model input capacity. That is, after forming candidate temporal segment intervals based on the legitimate candidate boundary translation, the historical observation part and the future action supervision part defined by the target action configuration parameters can be completely obtained from the robot task trajectory data, and the amount of data corresponding to the candidate temporal segment interval does not exceed the input capacity allowed by the video variational autoencoder or the vision-language-action model.

[0305] Next, based on the translation amounts of multiple legal candidate boundaries, the starting and ending boundary positions of the target time series segment intervals can be synchronously translated to obtain multiple candidate time series segment intervals that correspond one-to-one with each legal candidate boundary translation amount.

[0306] For any valid candidate boundary translation Based on the legal candidate boundary translation amount, the starting and ending boundary positions of the target time series segment interval can be synchronously translated in the same direction and by the same distance to obtain the corresponding candidate time series segment interval. : in, Indicates the translation amount of the legal candidate boundary. The corresponding candidate time series segmentation interval; Indicates the starting video frame index corresponding to the candidate time-series segment interval; This represents the index of the last video frame corresponding to the candidate time-series segment interval, and the index of the last video frame... It is not included in the candidate time series segment interval.

[0307] when When, it indicates that the target temporal segment interval is shifted backward along the time direction of the video frame sequence; when When, it indicates that the target temporal segment interval has been shifted forward along the time direction of the video frame sequence; when When, it means that the position of the target time series segment interval remains unchanged.

[0308] Since the starting and ending boundary positions of the candidate temporal segment intervals are synchronously translated using the same legal candidate boundary translation amount, the number of video frames contained in each candidate temporal segment interval is the same as the number of video frames contained in the target temporal segment interval before translation.

[0309] Furthermore, the second boundary evaluation results corresponding to each candidate time series segment interval can be determined.

[0310] The second boundary evaluation result is used to comprehensively characterize the degree of matching between the starting and ending boundary positions of each candidate temporal segment interval and the robot task execution process.

[0311] In one implementation, the second boundary evaluation result corresponding to the candidate time segment interval can be determined based on at least one of the motion cost, gripper state change cost, contact state change cost, state machine stage change cost, and translation distance cost corresponding to the start and end boundary positions of the candidate time segment interval.

[0312] The second boundary evaluation result corresponding to the candidate time series segment interval can be expressed as: in, Indicates the translation amount of the legal candidate boundary. The evaluation results of the second boundary of the corresponding candidate time series segment interval; This represents the motion cost corresponding to the starting boundary position of the candidate temporal segment interval, which is also the first boundary evaluation result; This represents the motion cost corresponding to the end boundary position of the candidate temporal segment interval, which is also the first boundary evaluation result; This indicates the cost of gripper state change when the boundary of the candidate temporal segment interval is located during the gripper state switching process; This represents the contact state change cost corresponding to the boundary of the candidate temporal segment interval being located during the contact state change process; The boundary of the candidate temporal segment interval represents the state machine stage change cost corresponding to the stage transition process of the robot state machine.

[0313] This represents the non-negative weight corresponding to the cost of the gripper state change. Represents the non-negative weights corresponding to the cost of changes in contact state; This represents the non-negative weight corresponding to the cost of state machine stage changes; This represents the non-negative weight corresponding to the translation distance cost; This represents the absolute value of the translation of the legitimate candidate boundary, used to characterize the translation distance of the candidate time series segment interval relative to the target time series segment interval before translation.

[0314] in, and It can be obtained by referring to the aforementioned implementation method.

[0315] The smaller the second boundary evaluation result, the smoother the robot's motion changes near the start and end boundary positions of the candidate temporal segment interval, and the fewer times the boundary of the candidate temporal segment interval falls during the transition between gripper state, contact state, or state machine stage. Therefore, candidate temporal segment intervals with smaller second boundary evaluation results are more suitable as independent training samples.

[0316] Conversely, the larger the second boundary evaluation result, the more obvious the changes in robot action, gripper state, contact state, or robot state machine stage are near the starting and / or ending boundary positions of the candidate time segment interval. Alternatively, the candidate time segment interval may have a large temporal position offset relative to the original target time segment interval. Therefore, the boundary position of the candidate time segment interval has a low degree of matching with the semantic stage of the robot task process.

[0317] Furthermore, the candidate time series segment interval with the smallest second boundary evaluation result can be determined as the translated second target time series segment interval.

[0318] Here, the set of multiple legal candidate boundary translations can be represented as: The translation amount of the legal candidate boundary corresponding to the candidate temporal segment interval with the smallest second boundary evaluation result. It can be represented as: in, This represents the evaluation result of the second boundary corresponding to the translation amount of the legitimate candidate boundary. This indicates the translation amount of the legal candidate boundary that minimizes the evaluation result of the second boundary.

[0319] Correspondingly, the shifted second target time series segmentation interval It can be represented as: in, This indicates the starting video frame index corresponding to the second target time-series segmentation interval; This indicates the index of the last video frame corresponding to the second target time-series segmentation interval, and the index of the last video frame... It is not included in the second target temporal segmentation interval, and the shifted second target temporal segmentation interval satisfies the temporal coding constraints of the video variational autoencoder.

[0320] In one implementation, when there is no valid candidate boundary translation amount that meets the requirements, zero can be determined as the optimal valid candidate boundary translation amount, so that the second target time series segment interval is consistent with the target time series segment interval before translation.

[0321] In another implementation, when the improvement of the second boundary evaluation result corresponding to the optimal legal candidate boundary translation amount relative to the second boundary evaluation result corresponding to the zero translation amount does not reach a preset threshold, zero can also be determined as the optimal legal candidate boundary translation amount, so as to avoid unnecessary translation of the target time series segment interval when the boundary optimization benefit is small.

[0322] In this way, when multiple target temporal segment intervals are used as independent training samples, the overall translation optimization can be performed on each target temporal segment interval separately, and the candidate temporal segment interval with the smallest second boundary evaluation result can be selected from multiple candidate temporal segment intervals as the second target temporal segment interval. In this way, while keeping the number of video frames and the temporal coding constraints of the video variational autoencoder unchanged, the boundary position of the independent training samples is made to better match the action change characteristics during the execution of the robot task.

[0323] In one optional implementation, when there is a continuous time series segment sequence composed of multiple target time series segment intervals in multiple target time series segment intervals, the segment boundary corresponding to the continuous time series segment sequence can be further optimized and adjusted globally.

[0324] When multiple target time-series segment intervals are used continuously as whole data, the boundary position between each target time-series segment interval not only affects the video content range of a single target time-series segment interval, but also affects the temporal continuity relationship between multiple target time-series segment intervals.

[0325] Because video variational autoencoders have temporal coding constraints, multiple consecutive target temporal segment intervals need to meet corresponding video frame number constraints and time alignment constraints. Therefore, independently adjusting only a single target temporal segment interval may result in overlapping or uncovered intervals between adjacent target temporal segment intervals, or inconsistencies in the overall temporal division relationship between adjacent target temporal segment intervals.

[0326] Therefore, for a continuous time series segment sequence consisting of multiple target time series segment intervals, the segment boundaries corresponding to the multiple target time series segment intervals can be optimized as a whole.

[0327] Specifically, the range of video frame indices corresponding to the continuous temporal segmentation sequence is determined.

[0328] The range of video frame indices corresponding to the continuous temporal segmentation sequence can be represented as: in, This indicates the range of video frame indices corresponding to a continuous temporal segment sequence. Indicates the starting video frame index of a segment sequence in a continuous temporal segment sequence; This indicates the end-of-segment video frame index of a continuous temporal segment sequence.

[0329] In one implementation, the range of video frame indices corresponding to a continuous temporal segment sequence can be the range of video frame indices corresponding to the entire video frame sequence, or it can be the range of video frame indices corresponding to a task stage.

[0330] When the task phase changes, or the target action configuration parameters or target temporal downsampling factor are reselected, the location of the configuration change can be determined as the starting point of the new continuous temporal segment sequence, and the global optimization process can be re-executed for the new continuous temporal segment sequence.

[0331] Furthermore, multiple candidate global segment boundary shifts can be determined based on the segmented sequence video frame index range.

[0332] Among them, the candidate global segment boundary translation is used to represent the translation amount of the boundary start position of the continuous time-series segment sequence.

[0333] Specifically, while keeping the number of video frames corresponding to each target temporal segmentation interval unchanged, multiple candidate continuous temporal segmentation sequences can be generated by changing the boundary start position of the continuous temporal segmentation sequence.

[0334] In one implementation, the candidate global segment boundary translation can satisfy the time alignment constraint corresponding to the target temporal downsampling factor.

[0335] For example, the candidate global segment boundary translation amount can preferably be an integer multiple of the target temporal downsampling factor.

[0336] The set of candidate global segment boundary translations can be represented as: in, Represents the set of candidate global segment boundary translations; Z represents a non-negative integer.

[0337] Because the difference is an integer number The video frame offsets correspond to the same continuous segmented grid relationship; therefore, only one... Search for candidate global segment boundary translations within the periodic range.

[0338] Furthermore, based on the preset legal constraints of global segment boundary translation, multiple candidate global segment boundary translations can be screened to obtain legal candidate global segment boundary translations.

[0339] Specifically, the legal constraints for global segmented boundary translations can include: (1) The number of candidate time series segment intervals contained in the candidate continuous time series segment sequence corresponding to the candidate global segment boundary translation amount is consistent with the number of original target time series segment intervals in the continuous time series segment sequence.

[0340] (2) The number of remaining frames at the beginning and the number of remaining frames at the end of the candidate continuous temporal segment sequence corresponding to the candidate global segment boundary translation amount meet the preset remaining frame limit.

[0341] (3) The candidate continuous temporal segment sequence corresponding to the candidate global segment boundary translation needs to ensure the integrity of historical observation information and the integrity of action supervision target.

[0342] (4) The candidate continuous temporal segment sequence corresponding to the candidate global segment boundary translation amount needs to satisfy the task stage boundary constraints.

[0343] (5) The candidate continuous temporal segment sequence corresponding to the candidate global segment boundary translation amount needs to satisfy the model input capacity constraint.

[0344] When the candidate continuous temporal segment sequence corresponding to the candidate global segment boundary translation satisfies the above constraints, the candidate global segment boundary translation can be determined as a valid candidate global segment boundary translation.

[0345] Furthermore, based on the translation amount of each legal candidate global segment boundary, the segment boundary corresponding to the continuous time series segment sequence is translated to obtain the corresponding candidate continuous time series segment sequence.

[0346] Specifically, for any valid candidate global segment boundary translation, the starting and ending boundary positions of multiple target time-series segment intervals in the continuous time-series segment sequence can be synchronously translated to obtain the corresponding candidate continuous time-series segment sequence.

[0347] In this method, the same legal candidate global segment boundary translation amount is used to perform overall translation of multiple target temporal segment intervals. Therefore, multiple candidate temporal segment intervals in each candidate continuous temporal segment sequence have the same number of video frames, and the boundaries between adjacent candidate temporal segment intervals remain continuous.

[0348] Furthermore, the global segmentation boundary evaluation results for each candidate continuous time-series segment can be determined.

[0349] Specifically, after determining the candidate continuous time series segment sequence, the internal boundary positions between adjacent candidate time series segment intervals in the candidate continuous time series segment sequence can be determined.

[0350] The internal boundary position is used to indicate the connection position between two adjacent candidate time series segment intervals.

[0351] The translation amount of the current legal candidate global segment boundary is: At that time, the set of candidate time series segment interval numbers that can be completely located within a continuous time series segment sequence can be represented as: in, This represents the set of candidate time series segment intervals whose numbers, after applying the candidate global segment boundary shift, can be completely located within the continuous time series segment sequence. Indicates the index of the last video frame in a continuous temporal segment sequence. It is a non-negative integer.

[0352] n is the total number of candidate time segment intervals that are completely located within the continuous time segment sequence.

[0353] The set of internal boundary position numbers between adjacent candidate time series segment intervals within the number set can be represented as: in, Let represent the set of internal boundary position numbers, where q represents the q-th internal boundary position in the set of internal boundary position numbers, and n-1 represents the total number of internal boundary positions.

[0354] No. The video frame index corresponding to each internal boundary position can be represented as: in, This indicates the use of valid candidate global piecewise boundary translations. The later formed The video frame index corresponding to each internal boundary.

[0355] Furthermore, the global segmentation boundary evaluation result corresponding to the candidate continuous temporal segmentation sequence can be determined based on the first boundary evaluation result corresponding to the internal boundary position between adjacent candidate temporal segmentation intervals that are completely located within the continuous temporal segmentation sequence, as well as the processing cost of the first and last remaining frames of the candidate continuous temporal segmentation sequence, that is, the video frames of the candidate temporal segmentation intervals that are not within the continuous temporal segmentation sequence.

[0356] Among them, the global segmented boundary evaluation results It can be represented as: in, Indicates the global segment boundary translation of a valid candidate. The global segmentation boundary evaluation results of the corresponding candidate continuous temporal segmentation sequences; Indicates the first The first boundary evaluation result corresponding to each internal boundary location; This indicates the use of valid candidate global piecewise boundary translations. Then, the processing cost of the first and last remaining frames corresponding to the candidate continuous temporal segmentation sequence; This represents the non-negative weight corresponding to the processing cost of the remaining frames at the beginning and end.

[0357] The smaller the global segment boundary evaluation result, the smoother the robot motion changes near the internal boundary position in the candidate continuous temporal segment sequence, and the smaller the processing cost corresponding to the first and last remaining frames.

[0358] Finally, the candidate continuous temporal segment sequence with the smallest global segmentation boundary evaluation result can be determined as the translated continuous temporal segment sequence.

[0359] The translation amount of the candidate global segment boundary corresponding to the translated continuous temporal segment sequence can be expressed as: in, This represents the final determined candidate global segment boundary translation amount; Represents the set of legal candidate global segment boundary translations; This indicates the candidate global segment boundary translation amount that minimizes the global segment boundary evaluation result.

[0360] That is, the candidate continuous temporal segment sequence corresponding to the final determined global segment shift is determined as the shifted continuous temporal segment sequence.

[0361] The translated continuous temporal segment sequence includes multiple target temporal segment intervals. Each target temporal segment interval maintains the same number of video frames, and adjacent target temporal segment intervals maintain continuous boundaries and satisfy the temporal coding constraints of the video variational autoencoder corresponding to the target temporal downsampling factor.

[0362] In one implementation, when there are remaining video frames at the beginning or end of a continuous temporal segment sequence that cannot form a complete target temporal segment interval, an effective frame mask can be used to locally fill in the gaps, generate a target temporal segment interval with a separate boundary, or directly truncate that part.

[0363] The above method can be used to optimize the overall boundary of multiple target time-series segment intervals used in succession, avoiding the problem of overlap or non-coverage between adjacent target time-series segment intervals caused by individual adjustments, and improving the consistency of video time-series segments corresponding to continuous robot tasks.

[0364] Step S206: For any target time series segment interval, determine whether there is a reusable latent variable corresponding to the target time series segment interval based on the latent variable reuse condition, and determine the target latent variable corresponding to the target time series segment interval based on the judgment result.

[0365] In step S206, after determining multiple target time series segment intervals, it can be determined whether there are reusable latent variables that can be directly reused for any target time series segment interval.

[0366] In some embodiments of this application, for the video frame sequence corresponding to the robot task trajectory data, there may be a temporal inclusion relationship or overlap relationship between different target time sequence segment intervals.

[0367] For example, a video temporal segment interval with a relatively long time range has been encoded by a video variational autoencoder and the corresponding video latent variables have been obtained. The target temporal segment interval generated later may be located inside the encoded video temporal segment interval.

[0368] Therefore, to avoid repeatedly performing video variational autoencoder encoding on the same video content, it is possible to determine whether the existing valid latent variables meet the reuse conditions before generating the target latent variables corresponding to the target temporal segment interval.

[0369] In one implementation, reference Figure 6 , Figure 6 This is an exploded flowchart of an embodiment of step S206 in the video latent variable generation method according to this application. Step S206 may include the following steps S2061-S2065.

[0370] Step S2061: Query at least one existing valid latent variable that has been encoded by the video variational autoencoder.

[0371] Specifically, it is possible to query all existing valid latent variables that have been encoded and saved by the video variational autoencoder.

[0372] Among them, the existing effective latent variables are the latent variables obtained after performing video variational autoencoder encoding based on historical video frame sequences.

[0373] In one implementation, existing effective latent variables can correspond to one or more time-series segment intervals that have been encoded, and can be used to determine whether the video latent variable generation requirements of the current target time-series segment interval can be met.

[0374] For example, if video variational autoencoder encoding has been performed on a time-series segmented interval over a relatively long time range for the robot's task trajectory data, the corresponding existing valid latent variables can be saved, and the saved existing valid latent variables can be queried when processing the robot's task trajectory data in the future.

[0375] In one implementation, associated information such as the temporal segment interval identifier, video frame index range, latent variable time index range, target temporal downsampling factor, and video variational autoencoder encoding conditions corresponding to existing effective latent variables can also be stored, so as to determine whether existing effective latent variables can be reused based on the associated information.

[0376] Step S2062: Determine the parent time series segment intervals for each existing effective latent variable.

[0377] Specifically, the time series segment intervals corresponding to each existing effective latent variable can be determined as the parent time series segment intervals of each existing effective latent variable.

[0378] The parent time-series segment interval is used to represent the range of video frame indices corresponding to existing valid latent variables.

[0379] The parent temporal segmentation interval and the target temporal segmentation interval can represent video segments corresponding to different video frame index ranges in the video frame sequence, respectively. The target temporal segmentation interval can be completely located within the parent temporal segmentation interval, or it can partially overlap with the parent temporal segmentation interval or not overlap at all.

[0380] For example, the parent time series segment interval can be represented as: in, This indicates the parent temporal segment interval for which video variational autoencoder encoding has already been performed; Indicates the starting video frame index corresponding to the parent time-series segment interval; This represents the index of the last video frame corresponding to the parent time-series segment interval, and the index of the last video frame... It is not included in the parent time-series segment interval.

[0381] Step S2063: Determine whether there exists a reusable parent time series segment interval that satisfies the latent variable reuse condition.

[0382] Each parent time series segment interval can be filtered based on the latent variable reuse condition, and the results of the filtering can be used to determine whether there are reusable parent time series segment intervals that meet the latent variable reuse condition.

[0383] Among them, the latent variable reuse condition is used to determine whether the parent time series segment interval corresponding to the existing valid latent variable meets the video latent variable reuse requirements corresponding to the current target time series segment interval.

[0384] In one implementation, the latent variable reuse condition includes at least one of the following: time interval coverage condition, time base consistency condition, time index mapping condition, time-series coding condition, context sufficiency condition, and coding condition consistency condition.

[0385] Among them, the time interval coverage condition is used to determine whether the parent time series segment interval covers the target time series segment interval.

[0386] Specifically, it can be determined whether the video frame index range corresponding to the target time segment interval is within the video frame index range corresponding to the parent time segment interval.

[0387] When the following conditions are met: When the target time segment interval is located, it can be determined that the video frame index range corresponding to the target time segment interval is within the video frame index range corresponding to the parent time segment interval, that is, it can be determined that the parent time segment interval covers the target time segment interval.

[0388] The time reference consistency condition is used to determine whether the parent time segment interval and the target time segment interval are determined based on the same original video frame time axis.

[0389] Specifically, it can be determined whether the video frames corresponding to the parent and target time-series segment intervals originate from the same video frame sequence in the same robot task trajectory data, and whether the video frame timestamp references used by the two are consistent. If the two are determined based on different video frame time axes or different timestamp references, it cannot be guaranteed that there is a consistent correspondence between the latent variable time index of the existing valid latent variables and the target time-series segment interval.

[0390] The time index mapping condition is used to determine whether the offset of the starting position of the target time series segment interval relative to the parent time series segment interval meets the time index mapping requirements of the target time series downsampling factor.

[0391] Specifically, it can be determined whether the offset of the starting position of the target time series segment interval relative to the parent time series segment interval, and the length of the target time series segment interval, both satisfy the divisibility requirement of the target time series downsampling factor. If satisfied, the target time series segment interval can be mapped to the integer time index in the parent effective latent variable; if not satisfied, latent variable slicing cannot be directly performed.

[0392] Wherein, the starting position of the target time series segment interval relative to the starting position of the parent time series segment interval can satisfy: in, This represents the video frame offset of the starting video frame index of the target temporal segment interval relative to the starting video frame index of the parent temporal segment interval. Indicates the starting video frame index of the target temporal segmentation interval; Indicates the starting video frame index of the parent time-series segment interval.

[0393] when When the starting video frame index of the target time segment interval is divisible by the starting video frame index of the parent time segment interval, it means that the starting video frame index of the target time segment interval satisfies the integer division requirement of the target time downsampling factor relative to the starting video frame index of the parent time segment interval. This allows the starting video frame index of the target time segment interval to be mapped to the integer time index of the latent variable in the existing valid latent variable corresponding to the parent time segment interval, and it will not fall into the video frame range corresponding to a certain latent variable time step.

[0394] Furthermore, the number of video frames contained in the target temporal segment interval can also satisfy: in, Indicates the number of video frames contained in the target temporal segment interval; This represents the index of the last video frame in the target temporal segmentation interval, and the index of the last video frame... It is not included in the target time series segmentation interval.

[0395] when When, it means that the number of video frames contained in the target time segment interval can meet the divisibility requirement of the target time downsampling factor, so that the complete latent variable time step can be extracted from the existing effective latent variables corresponding to the parent time segment interval.

[0396] Furthermore, the starting frame offset of the target temporal segment interval relative to the parent temporal segment interval. satisfy: When the target time segment interval is determined relative to the parent time segment interval, the starting frame offset can be determined, that is, the starting position offset satisfies the time index mapping condition corresponding to the target time downsampling factor.

[0397] Specifically, when the starting position of the target time-series segment interval is offset from the starting position of the parent time-series segment interval by an integer number of video frames corresponding to the target time-series downsampling factor, the starting video frame position of the target time-series segment interval can be mapped to the integer latent variable time index corresponding to the parent time-series segment interval.

[0398] The timing coding condition is used to determine whether the parent timing segment interval and the target timing segment interval use the same video variational autoencoder target timing downsampling factor.

[0399] When the target time-series downsampling factor corresponding to the parent time-series segment interval is consistent with that of the target time-series segment interval, it can be determined that the time-series coding conditions are met.

[0400] In one implementation, the timing coding condition can also be used to determine whether the number of video frames in the parent timing segmentation interval and the target timing segmentation interval respectively meet the corresponding video variational autoencoder timing coding constraints, so as to avoid extracting the target latent variable from existing valid latent variables that do not meet the timing coding requirements.

[0401] The context sufficiency condition is used to determine whether the parent temporal segment interval can provide context information that meets the temporal coding requirements of the video variational autoencoder for the target temporal segment interval.

[0402] When generating latent variables near the start or end of the target temporal segmentation interval, the video variational autoencoder may need to use historical or future video frames outside the target temporal segmentation interval as context information.

[0403] Therefore, even if the parent time series segmentation interval covers the target time series segmentation interval, and the target time series segmentation interval can be mapped to the integer latent variable time index corresponding to the parent time series segmentation interval, it is still necessary to determine whether the left and right contexts provided by the encoded input corresponding to the parent time series segmentation interval are sufficient.

[0404] The available left-hand context that the parent temporal segment interval can provide for the starting video position of the target temporal segment interval can be represented as: in, This indicates the number of video frames that are actually available for the video variational autoencoder before the parent temporal segmentation interval begins at the target temporal segmentation interval. This indicates the number of additional context video frames added to the left of the parent temporal segment interval when performing video variational autoencoder encoding on the parent temporal segment interval; This represents the video frame offset of the starting video frame position of the target temporal segment interval relative to the starting video frame position of the parent temporal segment interval.

[0405] Therefore, the closer the starting video frame position of the target time segment interval is to the inside of the parent time segment interval, the more available left-hand context the parent time segment interval can provide for the starting video frame position of the target time segment interval.

[0406] The available right-hand context that the parent time series segment interval can provide for the end position of the target time series segment interval can be represented as: in, This indicates the number of video frames that are actually available for the video variational autoencoder after the parent temporal segmentation interval is located at the end of the target temporal segmentation interval. This indicates the number of additional context video frames added to the right of the parent temporal segment interval when performing video variational autoencoder encoding on the parent temporal segment interval; This indicates the number of video frames between the end video frame position of the target time segment interval and the end video frame position of the parent time segment interval.

[0407] When a video variational autoencoder employs a bidirectional temporal structure, the generation of latent variables depends simultaneously on the video input from both the historical and future directions. Therefore, the left-hand context can satisfy: And the context on the right can satisfy in, This represents the range of valid input video frames required in the historical direction when the video variational autoencoder generates latent variables near the starting position of the target temporal segment interval; This represents the range of valid input video frames required in the future direction when the video variational autoencoder generates latent variables near the end position of the target temporal segment interval. and All figures are in units of video frames.

[0408] When both of the above formulas are satisfied, it can be determined that the parent time series segment interval can provide sufficient bidirectional time series context for the target time series segment interval. The target latent variable obtained by slicing from the existing effective latent variables corresponding to the parent time series segment interval matches the context conditions when encoding the target time series segment interval alone.

[0409] When a video variational autoencoder employs a causal temporal structure, the current latent variable does not depend on the video input in future directions. Therefore, it is sufficient to determine whether the historical context is adequate, i.e., only the formula needs to be evaluated. It holds true, without requiring a formula. Established.

[0410] In one implementation, when the video variational autoencoder adopts other temporal structures, the context direction to be checked and the corresponding number of context video frames can be determined based on the effective temporal receptive field range corresponding to the video variational autoencoder.

[0411] The coding condition consistency condition is used to determine whether the coding conditions of the video variational autoencoder corresponding to the parent temporal segment interval and the target temporal segment interval are consistent.

[0412] The video variational autoencoder encoding conditions include at least one of the following: video variational autoencoder model parameters, temporal encoding parameters, video preprocessing parameters, and encoding process parameters.

[0413] For example, it is possible to compare whether the video variational autoencoder model version, model weights, temporal coding parameters, video preprocessing methods, input video frame resolution, image normalization methods, camera sources, and video frame order corresponding to the parent temporal segment interval and the target temporal segment interval are consistent.

[0414] In one implementation, the parent time series segment interval can be determined to satisfy the latent variable reuse condition when it simultaneously satisfies the time interval coverage condition, the time base consistency condition, the time index mapping condition, the time series coding condition, the context sufficiency condition, and the coding condition consistency condition.

[0415] In another implementation, some of the above conditions can be selected as latent variable reuse conditions based on the specific video variational autoencoder structure and latent variable storage method.

[0416] Step S2064: When there is a reusable parent time series segment interval, determine that there is a reusable latent variable corresponding to the target time series segment interval.

[0417] Specifically, when there is a reusable parent time series segment interval, the existing effective latent variable corresponding to the reusable parent time series segment interval can be determined as a reusable latent variable of the target time series segment interval.

[0418] Step S2065: When there is no reusable parent time series segmentation interval, determine that there is no reusable latent variable corresponding to the target time series segmentation interval.

[0419] Specifically, when all parent time series segment intervals do not meet the latent variable reuse condition, that is, when there are no reusable parent time series segment intervals, it can be determined that there are no reusable latent variables corresponding to the target time series segment interval.

[0420] At this point, the target time-series segment interval needs to undergo a further video latent variable generation process.

[0421] By using the above method, the matching relationship between existing effective latent variables and target time series segment intervals can be judged before generating target latent variables in the target time series segment intervals, thereby determining whether there are reusable latent variables in the current target time series segment intervals.

[0422] This approach avoids re-performing the latent variable generation judgment for all target time-series segment intervals, thus improving the processing efficiency of subsequent video latent variable generation processes.

[0423] In one alternative implementation, for the same robot task trajectory data, coarse-grained time series segmentation intervals and fine-grained time series segmentation intervals can also be established according to different time granularities.

[0424] Among them, the range of coarse-grained temporal segment video frame indexes corresponding to the coarse-grained temporal segment intervals is greater than the range of fine-grained temporal segment video frames corresponding to the fine-grained temporal segment intervals.

[0425] In other words, the video frame coverage area corresponding to a coarse-grained temporal segment interval is larger than that corresponding to a fine-grained temporal segment interval. Coarse-grained temporal segment intervals can be used to characterize the robot task process over a longer time range, while fine-grained temporal segment intervals can be used to characterize local action processes over a shorter time range.

[0426] When a fine-grained time series segmentation interval is completely covered by a single coarse-grained time series segmentation interval, the fine-grained time series segmentation latent variable corresponding to the fine-grained time series segmentation interval can be determined based on the coarse-grained time series segmentation latent variable corresponding to the coarse-grained time series segmentation interval.

[0427] Furthermore, it can be determined whether the fine-grained time series segmentation interval and the coarse-grained time series segmentation interval satisfy the time index mapping condition and the time series coding condition in the latent variable reuse condition.

[0428] When a fine-grained time series segment interval is completely covered by a single coarse-grained time series segment interval, and the fine-grained time series segment interval and the coarse-grained time series segment interval satisfy the time index mapping condition and the time series coding condition in the latent variable reuse condition, the target latent variable corresponding to the fine-grained time series segment interval can be determined based on the latent variable corresponding to the coarse-grained time series segment interval.

[0429] When a fine-grained temporal segmentation interval crosses the boundary position between multiple coarse-grained temporal segmentation intervals, video variational autoencoder encoding is performed on the fine-grained temporal segmentation interval to obtain the fine-grained temporal segmentation latent variables corresponding to the fine-grained temporal segmentation interval.

[0430] Furthermore, when the time index mapping condition and / or timing coding condition in the latent variable reuse condition are not satisfied between the fine-grained temporal segmentation interval and the coarse-grained temporal segmentation interval, the target latent variable corresponding to the fine-grained temporal segmentation interval is obtained by performing context-based video variational autoencoder encoding based on the video frame sequence corresponding to the fine-grained temporal segmentation interval.

[0431] Specifically, context video frames that meet the effective temporal receptive field requirements of a video variational autoencoder can be added to the left and / or right of the fine-grained temporal segmentation interval. The obtained extended video frame sequence is then encoded using a video variational autoencoder, and the context latent variables corresponding to the context video frames in the encoding result are removed, thereby obtaining the fine-grained temporal segmentation latent variables corresponding to the fine-grained temporal segmentation interval.

[0432] Step S207: When there are reusable latent variables that meet the latent variable reuse conditions, the target latent variables for the target time series segment interval are obtained based on the reusable latent variables.

[0433] This step S207 is used to obtain the target latent variable for the corresponding target time series segment interval based on the reusable latent variable when there is a reusable latent variable that satisfies the latent variable reuse condition.

[0434] In one optional implementation, the time index range of the reusable latent variable corresponding to the reusable latent variable can be determined based on the time offset of the target time series segment interval relative to the reusable parent time series segment interval. Then, based on the time index range of the reusable latent variable, time dimension slicing processing is performed on the reusable latent variable to obtain the target latent variable corresponding to the target time series segment interval.

[0435] For example, suppose the reusable effective latent variable corresponding to the reusable parent time series segment interval is... The video variational autoencoder uses a target temporal downsampling factor. Under the condition of uniform temporal downsampling and satisfying the temporal coding constraints of the video variational autoencoder, the effective latent variable step size in the time dimension can be reused. It can be represented as: The above formula is applicable to able to be It is divisible, and the video variational autoencoder does not have special processing for the first frame, asymmetric time padding, or other non-simple time mapping.

[0436] The latent variable start time index of the latent variable in the target time series segment interval within the reusable latent variables can be represented as: in, This represents the starting time index of the target latent variable in the time dimension of the reusable latent variables, and It is a non-negative integer; This represents the video frame offset of the starting video frame position of the target temporal segment interval relative to the starting video frame position of the reusable parent temporal segment interval.

[0437] The step size of the latent variable in the time dimension corresponding to the target time series segment interval It can be represented as: The latent variable end time index of the latent variable in the target time series segment interval within the reusable latent variables can be represented as: in, This represents the latent variable end-time index of the target latent variable slice. Since the latent variable slice uses a left-closed, right-open latent variable time index format, The time step of the latent variable corresponding to the location is not included in the target latent variable.

[0438] The latent variable end time index of the latent variable in the target time series segment interval within the reusable latent variables can also be represented as: Furthermore, it can be indexed based on the start time of latent variables. and latent variable end time index Extract the target latent variables corresponding to the target time series segment intervals from the reusable latent variables: Among them, target latent variables It can be represented as: in, This represents the target latent variable corresponding to the target time series segment interval; Represents the reusable latent variable corresponding to the reusable parent time series segment interval; symbol This indicates performing a left-closed, right-open slicing operation along the time dimension of the reusable latent variable, i.e., preserving the time index. to The corresponding latent variable time step. During slicing, the spatial dimension and channel dimension of reusable latent variables remain unchanged.

[0439] When performing latent variable slicing, the slice index can satisfy: in, This indicates that the starting point of the slice for the target latent variable is no earlier than the starting position of the reusable latent variable; This indicates that the target latent variable contains at least one latent variable time step; This indicates that the slice endpoint of the target latent variable does not exceed the time index range of the reusable latent variable.

[0440] The actual slice length of the target latent variable can also satisfy: in, This represents the actual number of latent variable time steps extracted from the reusable latent variables; This represents the theoretical number of latent variable time steps, determined based on the number of original video frames contained within the target temporal segment interval. This formula can be used to verify whether the actual latent variable slice length matches the latent variable time length corresponding to the target temporal segment interval.

[0441] For example, suppose the reusable parent time series segment interval is The target time series segmentation interval is Target time series downsampling factor The value is 4.

[0442] In this example, the target temporal segment interval is completely covered by the reusable parent temporal segment interval; the starting video frame position of the target temporal segment interval is offset by 8 frames relative to the starting video frame position of the parent temporal segment interval; the target temporal segment interval contains 20 video frames. Since both 8 and 20 are divisible by the target temporal downsampling factor 4, the target temporal segment interval can be mapped to the integer latent variable time index in the reusable latent variables.

[0443] Accordingly, it can be determined It is 2. It is 5. It is 7.

[0444] in, =2 indicates that the target latent variable starts from the second latent variable time index of the reusable latent variables; =5 indicates that the target time series segmentation interval corresponds to 5 latent variable time steps; =7 indicates that the right-hand end index of the latent variable slice is 7.

[0445] Therefore, we can start from reusable latent variables. Extract from the middle: The slice is the target latent variable corresponding to the target time series segment interval. The slice actually includes latent variable time indices 2, 3, 4, 5 and 6, a total of 5 latent variable time steps, and does not include the latent variable time step corresponding to the right end index 7.

[0446] For example, if the target time series segmentation interval is If the starting video frame position of the target time-series segmented interval is offset by 10 frames relative to the starting video frame position of the parent time-series segmented interval, then since 10 is not divisible by the target time-series downsampling factor 4, the starting video frame position of the target time-series segmented interval cannot be mapped to the integer latent variable time index in the existing valid latent variables.

[0447] In this case, precise temporal slicing should not be performed directly on the target temporal segment interval from the existing valid latent variables. Instead, the target temporal segment interval that satisfies the temporal index mapping condition can be redefined, or the target temporal segment interval can be encoded using a video variational autoencoder with temporal context (described later) to obtain the corresponding target latent variables.

[0448] When a video variational autoencoder does not satisfy a simple uniform time mapping, for example, when the video variational autoencoder has special first-frame processing, asymmetric time padding, or other non-simple time mapping, the same time boundary mapping function as the video variational autoencoder encoding process with temporal context described later can be used. Calculate the start time index and end time index of the latent variable in the reusable latent variable for the target time series segment interval.

[0449] Step S208: When there are no reusable latent variables that meet the latent variable reuse conditions, perform video variational autoencoder encoding on the video frame sequence of the target temporal segment interval to obtain the target latent variables of the target temporal segment interval.

[0450] In step S208, when step S206 determines that there are no reusable latent variables that meet the latent variable reuse conditions, the video variational autoencoder encoding can be re-executed for the corresponding target temporal segment interval to obtain the corresponding target latent variable.

[0451] In some embodiments of this application, when existing effective latent variables cannot meet the latent variable reuse conditions of the target temporal segmentation interval, video variational autoencoder encoding can be directly performed based on the video frame sequence corresponding to the target temporal segmentation interval to generate target latent variables corresponding to the target temporal segmentation interval.

[0452] In one implementation, reference Figure 7 , Figure 7 This is an exploded flowchart of an embodiment of step S208 in the video latent variable generation method according to this application. Step S208 may include the following steps S2081-S2085.

[0453] Step S2081: Obtain the effective temporal receptive field range of the video variational autoencoder.

[0454] Because the target latent variable may depend not only on the video frames within the target temporal segment interval during the encoding process of the video variational autoencoder, but also on the video context information of adjacent time ranges, it is necessary to add context video frames to the left and / or right of the target temporal segment interval.

[0455] Specifically, when there are no reusable latent variables corresponding to the target temporal segmentation interval, the effective temporal receptive field range of the video variational autoencoder can be obtained.

[0456] The effective temporal receptive field range is used to represent the range of video frames that the video variational autoencoder needs to reference in the historical and / or future directions when generating the target latent variables corresponding to the target temporal segment interval.

[0457] Step S2082: Based on the effective temporal receptive field range, add context video frames to the left and / or right of the target temporal segmentation interval to generate an extended video frame sequence, and determine the context video frame index range corresponding to the context video frames in the extended video frame sequence.

[0458] In this embodiment, the number of left and / or right context video frames required for encoding the target temporal segment interval can be determined based on the effective temporal receptive field range of the video variational autoencoder. Corresponding context video frames are then added to the left and / or right of the target temporal segment interval according to the number of left and / or right context video frames, thereby generating an extended video frame sequence.

[0459] The context video frame is used to provide additional temporal information to the latent variables corresponding to the edge positions of the target temporal segment interval when performing local coding on the video variational autoencoder, thereby reducing the boundary effect caused by the truncation of the local coding boundary. Adding a context video frame only expands the actual input range of the video variational autoencoder and does not change the start and end video frame positions or the length of the temporal segment interval itself.

[0460] In one implementation, the effective temporal receptive field range of the video variational autoencoder in the historical and future directions can be determined first.

[0461] in, This represents the range of valid input video frames required in the historical direction when the video variational autoencoder generates latent variables near the starting video frame position of the target temporal segment interval; This represents the range of valid input video frames required in the future direction when the video variational autoencoder generates latent variables near the end of the target temporal segment interval video frame. and All figures are in units of video frames.

[0462] The effective temporal receptive field range corresponding to the video variational autoencoder can be determined based on at least one of the temporal convolution kernel size, number of network layers, time step, dilation rate, or local attention window of the video variational autoencoder.

[0463] In another implementation, the coding results of a local video variational autoencoder can also be compared with the coding results obtained by continuously coding a longer video frame sequence to determine the encoding outcome. and Perform experimental calibration.

[0464] When the video variational autoencoder adopts a bidirectional timing structure with symmetrical front-to-back action, we can set: in, The effective temporal receptive field radius of the video variational autoencoder is the maximum range of video frames in which the historical and future directions can effectively influence the latent variable at any position in the target temporal segment interval.

[0465] After determining the effective temporal receptive field range, the actual number of additional context video frames that need to be added can be determined by combining the preset minimum context length.

[0466] Let the preset minimum number of left-side context video frames be... The preset minimum number of right-side context video frames is The actual number of left-side context video frames can be expressed as: in, This indicates the actual number of original video frames added to the left of the target time segment interval; Indicates the preset minimum number of left-side context video frames; This represents the effective input video frame range required by the video variational autoencoder in the historical direction. Using the above formula, the actual number of left-side context video frames can be ensured to be no less than the preset minimum number of left-side context video frames and the effective input range required by the video variational autoencoder in the historical direction, and... Target time series downsampling factor Integer multiples of.

[0467] The actual number of right-side context video frames can be expressed as: in, This indicates the actual number of original video frames added to the right of the target time segment interval; Indicates the preset minimum number of right-side context video frames; This represents the range of valid input video frames required by the video variational autoencoder in the future direction.

[0468] Using the above formula, the actual number of right-side context video frames can be ensured to be no less than the preset minimum number of right-side context video frames and the effective input range required by the video variational autoencoder in the future direction, and... Target time series downsampling factor Integer multiples of.

[0469] When a video variational autoencoder adopts a bidirectional temporal structure, context video frames can be added simultaneously to the left and right sides of the target temporal segment interval.

[0470] When a video variational autoencoder employs a causal temporal structure, the latent variable corresponding to the current position depends only on the current position and the video input in its historical directions, and not on the video input in future directions. Therefore, it can be set... It is 0, and historical context video frames are added only to the left of the target temporal segmentation interval.

[0471] In online inference scenarios, video frames that have not been acquired after the current moment are not used as right-side context video frames to avoid introducing future information during the generation of target latent variables.

[0472] Furthermore, the time range corresponding to the extended video frame sequence can be determined based on the target time sequence segmentation interval, the actual number of left-side context video frames, and the actual number of right-side context video frames.

[0473] Let the target time series segmentation interval be... The range of video frame indices corresponding to the extended video frame sequence can then be represented as: in, This indicates the range of video frame indices corresponding to the extended video frame sequence. Indicates the index of the starting video frame corresponding to the extended video frame sequence; This indicates the index of the end video frame corresponding to the extended video frame sequence, and this end video frame index is not included in the time range corresponding to the extended video frame sequence.

[0474] The extended video frame sequence includes the video frames corresponding to the target temporal segmentation interval, as well as the context video frames located to the left and / or right of the target temporal segmentation interval.

[0475] The total number of video frames contained in the extended video frame sequence can be expressed as: in, This represents the total number of video frames in the target temporal segmentation interval of the actual input video variational autoencoder. Indicates the length of the time series segment interval corresponding to the target time series segment interval; and These represent the actual number of video frames in the left context and the actual number of video frames in the right context, respectively.

[0476] The length of the time series segment interval corresponding to the target time series segment interval. Actual number of left-side context video frames and the actual number of video frames in the right-side context. All are target time series downsampling factors When the total number of video frames in the extended video frame sequence is an integer multiple of the given value, the following condition can be met: It should be noted that the above formula is only used to indicate that the input length of the extended video frame sequence satisfies the integer division constraint corresponding to the target temporal downsampling factor, and does not limit the time dimension length of the output latent variable of any video variational autoencoder to be equal to the integer division constraint. When a video variational autoencoder has special first-frame processing, asymmetric time padding, or other non-simple time mapping rules, the time dimension length of the output latent variables can be determined according to the actual time coding rules corresponding to the video variational autoencoder.

[0477] When the time range corresponding to the extended video frame sequence exceeds the starting or ending video frame position of the original video frame sequence, the missing context video frames can be supplemented by boundary video frame copying, reflection filling, preset constant filling, or other boundary processing methods, and the corresponding valid frame mask can be generated.

[0478] The effective frame mask is used to distinguish between real video frames in the extended video frame sequence and supplementary video frames generated through boundary processing.

[0479] In one implementation, when the target temporal segmentation interval is located at the boundary of the original video frame sequence and there are insufficient real context video frames available, it can be determined whether the target latent variable corresponding to the target temporal segmentation interval meets the preset quality requirements based on the number of supplementary video frames or the completeness of the context.

[0480] When the context information obtained by supplementing video frames is insufficient to meet the preset quality requirements, it is possible to prohibit the target latent variables subsequently generated in the target time series segment interval from being used as reusable latent variables for other target time series segment intervals.

[0481] Using the above method, the context requirements can be determined first based on the effective temporal receptive field range of the video variational autoencoder. Then, the number of context video frames can be time-aligned according to the target temporal downsampling factor, and corresponding context video frames can be added to the left and / or right of the target temporal segmentation interval, thereby obtaining an extended video frame sequence that meets the encoding requirements of the local video variational autoencoder.

[0482] Additionally, the range of context video frame indices corresponding to context video frames in the extended video frame sequence can be determined.

[0483] Step S2083: Perform video variational autoencoder encoding on the extended video frame sequence to obtain the initial latent variables of the target temporal segmentation interval.

[0484] Specifically, after obtaining the extended video frame sequence, the extended video frame sequence can be input into a video variational autoencoder to uniformly encode the target temporal segmentation interval and its left and / or right context video frames to obtain the corresponding initial latent variables of the target temporal segmentation interval.

[0485] The extended video frame sequence includes the original video frames corresponding to the target temporal segmentation interval and context video frames added to reduce local coding boundary effects. During the encoding process, the video variational autoencoder can utilize the historical and / or future temporal information provided by the context video frames to generate latent variables corresponding to the edge positions of the target temporal segmentation interval.

[0486] The initial latent variables obtained by performing video variational autoencoder encoding on the extended video frame sequence can be expressed by the following formula: in, This indicates the encoder corresponding to the video variational autoencoder; This indicates the range of video frame indices corresponding to the extended video frame sequence. This represents the initial latent variables obtained after performing video variational autoencoder encoding on the extended video frame sequence.

[0487] It should be noted that in this formula... This is used to represent the range of video frame indices corresponding to an extended video frame sequence. In the actual encoding process, video frames within this time range can be extracted from the original video frame sequence and used as input to a video variational autoencoder.

[0488] When the range of video frame indexes corresponding to the extended video frame sequence exceeds the starting or ending video frame position of the original video frame sequence, the supplementary video frames obtained in step S2082 through boundary video frame copying, reflection filling, preset constant filling, or other boundary processing methods can be input into the video variational autoencoder.

[0489] In one implementation, the effective frame mask corresponding to the extended video frame sequence can be associated with the extended video frame sequence and input into the video variational autoencoder, or the effective frame mask can be retained before and after encoding to identify the time position in the initial latent variables that may be affected by the supplementary video frames.

[0490] Initial latent variables It can include temporal dimension, spatial dimension, and channel dimension. The temporal dimension is used to characterize the encoding result of the extended video frame sequence in the temporal direction; the spatial dimension is used to characterize the features of the video content in the spatial direction; and the channel dimension is used to characterize the different latent variable feature channels.

[0491] When subsequently removing contextual latent variables based on the time range corresponding to the context video frames, it is possible to proceed only along the initial latent variables. The time dimension is sliced ​​while the spatial and channel dimensions remain unchanged.

[0492] When uniform temporal downsampling is used in the video variational autoencoder, and the input video frame boundary and the latent variable temporal boundary satisfy a simple divisibility relationship, the number of time steps of the initial latent variable in the time dimension can be determined based on the number of video frames contained in the extended video frame sequence and the target temporal downsampling factor.

[0493] For example, when the extended video frame sequence contains a total number of video frames Able to be sampled by the target time series downsampling factor When the data is divisible and the video variational autoencoder does not have special first-frame processing, asymmetric time padding, or other non-simple time mapping, the time step number corresponding to the initial latent variable can be: The extended video frame sequence satisfies: This only indicates that the input length of the extended video frame sequence meets the time alignment requirement corresponding to the target temporal downsampling factor; it does not necessarily mean that the initial latent variable time step of any video variational autoencoder output is equal to... .

[0494] When a video variational autoencoder has special first-frame processing, asymmetric time padding, causal convolution, non-integer output length rules, or other non-simple time mapping structures, the temporal dimension length of the initial latent variables and the mapping relationship between the temporal boundaries of each video in the extended video frame sequence and the temporal boundaries of each latent variable in the initial latent variables can be determined according to the actual network structure and time coding rules of the video variational autoencoder.

[0495] In one implementation, the model version of the video variational autoencoder, the target temporal downsampling factor, the time padding method, the video preprocessing parameters, and the temporal boundary mapping rules can be compared with the initial latent variables. Save as associated.

[0496] Using the above method, a unified video variational autoencoder can be used to encode the target temporal segment interval and its necessary context video frames, so that the obtained initial latent variables simultaneously contain the latent variables corresponding to the target temporal segment interval and the context latent variables corresponding to the context video frames, providing a data foundation for subsequent removal of context latent variables and obtaining target latent variables.

[0497] Step S2084: Based on the context video frame index range and the target temporal downsampling factor, determine the context latent variables corresponding to the context video frames in the initial latent variables, and remove the context latent variables.

[0498] After obtaining the initial latent variables corresponding to the extended video frame sequence, the context latent variables corresponding to the context video frames in the initial latent variables can be determined based on the context video frame index range and the target temporal downsampling factor, and then the context latent variables can be removed.

[0499] In one implementation, the target time series segment interval may be determined to be within the time index range of the initial latent variables, and the initial latent variables corresponding to the target time series segment interval within the time index range of the initial latent variables may be retained, so as to remove the context latent variables in the initial latent variables.

[0500] Among them, the process of removing contextual latent variables only applies to the time dimension of the initial latent variables, while the spatial and channel dimensions of the initial latent variables remain unchanged.

[0501] In one implementation, when the video variational autoencoder uses uniform temporal downsampling and the input video frame boundary and the latent variable time boundary satisfy a simple divisibility relationship, the target temporal segment interval can be determined within the latent variable time index range of the initial latent variable based on the number of left-side context video frames, the length of the temporal segment interval, and the target temporal downsampling factor.

[0502] The latent variable time steps corresponding to the left-side context video frame can be expressed as: in, This indicates the time step of the left-hand context latent variable that precedes the corresponding latent variable in the target time series segment interval among the initial latent variables; This indicates the actual number of left-side context video frames determined in step S2082; This represents the target time-series downsampling factor.

[0503] Due to the actual number of left-side context video frames Target time series downsampling factor Therefore, it is an integer multiple of , It is a non-negative integer.

[0504] The effective latent variable time steps corresponding to the target time series segment interval can be expressed as: in, This represents the number of latent variable time steps corresponding to the target time series segment interval; This indicates the length of the time series segment interval corresponding to the target time series segment interval.

[0505] Due to the length of the time series segmentation interval The time alignment constraint corresponding to the target time series downsampling factor is satisfied, therefore, It is a positive integer.

[0506] Furthermore, it can be Determine the latent variable start time index of the latent variable corresponding to the target time series segment interval in the time dimension of the initial latent variable, and then... The latent variable end time index is determined as the latent variable corresponding to the target time series segment interval in the time dimension of the initial latent variable.

[0507] Accordingly, the remaining latent variables retained from the initial latent variables can be expressed as: in, This represents the remaining latent variables obtained after removing the context latent variables from the initial latent variables; Represents the initial latent variables obtained in step S2083; symbol This indicates that a left-closed, right-open slicing operation is performed along the time dimension of the initial latent variable.

[0508] Specifically, the slicing operation preserves the latent variable time index. Latent variable time index The corresponding latent variable time step, excluding the latent variable end time index. The corresponding latent variable time step.

[0509] By As the starting time index of latent variables, the contextual latent variables corresponding to the left-hand contextual video frames preceding the target temporal segment interval can be removed. This is achieved by limiting the slice length to... This allows the contextual latent variables corresponding to the right-side context video frame to be naturally outside the retention range, thereby removing the right-side contextual latent variables at the same time.

[0510] The length of the remaining latent variables obtained after removing context latent variables in the time dimension can satisfy the following: in, It represents the length of the target latent variable in the time dimension.

[0511] This formula is used to verify the consistency between the number of latent variable time steps contained in the remaining latent variables and the number of original video frames contained in the target temporal segment interval as well as the target temporal downsampling factor.

[0512] The left and right context video frames are used only during the video variational autoencoder encoding process to provide historical and / or future temporal information for the generation of latent variables corresponding to the target temporal segment interval. The contextual latent variables corresponding to the left and right context video frames are not retained as valid latent variables corresponding to the current target temporal segment interval.

[0513] In another implementation, when the video variational autoencoder has special first-frame processing, asymmetric time padding, causal convolution, non-integer output length rules, or other non-simple time mapping structures, the input video frame position and the latent variable time index may not satisfy a simple mapping relationship divided by the target temporal downsampling factor.

[0514] In this case, the temporal boundary mapping function corresponding to the video variational autoencoder can be used to determine the start and end indices of the target temporal segment interval in the initial latent variable time dimension.

[0515] The time dimension index range of the latent variables corresponding to the target time series segment interval can be represented as: in, This represents the temporal boundary mapping function corresponding to the video variational autoencoder; and These represent the starting and ending video frame indices of the target temporal segmentation interval, respectively. and These represent the starting and ending video frame indices of the extended video frame sequence, respectively. The starting video frame index of the target temporal segmentation interval corresponds to the starting time index of the latent variable in the initial latent variable time dimension; This represents the latent variable end time index corresponding to the end video frame index of the target temporal segment interval in the initial latent variable time dimension.

[0516] Time boundary mapping function The network structure, time padding method, timing downsampling path, and output time length rules of the video variational autoencoder can be pre-calculated or experimentally calibrated and associated with the corresponding video variational autoencoder encoding conditions.

[0517] Furthermore, based on the start and end time indices of the latent variables, a time-dimensional slice can be performed on the initial latent variables to obtain the remaining latent variables after removing the context latent variables: The slice intervals are arranged in a left-closed, right-open format, retaining the time index. To Time Index The corresponding latent variable time step, excluding the time index. The corresponding latent variable time step.

[0518] Before performing the slicing operation, the start time index and end time index of the latent variables can be validated to ensure that the start index is not less than zero, the end time index of the latent variables is greater than the start time index of the latent variables, and the end time index of the latent variables does not exceed the length of the initial latent variables in the time dimension.

[0519] By using the above method, the corresponding range of the target temporal segment interval in the time dimension of the initial latent variable can be accurately determined, and the context latent variable corresponding to the context video frame can be removed, thereby obtaining the remaining latent variable that corresponds only to the target temporal segment interval.

[0520] Step S2085: The remaining latent variables after removing contextual latent variables are determined as the target latent variables for the target time series segment interval.

[0521] Specifically, the remaining latent variables after removing the context latent variables corresponding to the context video frames can be determined as the target latent variables corresponding to the current target time-series segment interval.

[0522] Among them, the target latent variables only correspond to the video content of the target time-series segment interval and do not include the latent variable information corresponding to the context video frames.

[0523] For example, let the target time series segmentation interval be... Time series segmentation interval length The target time-series downsampling factor is 20. The video variational autoencoder has a bidirectional temporal structure with a value of 4, and adds 4 context video frames to the left and right sides of the target temporal segmentation interval.

[0524] At this point, the time range corresponding to the extended video frame sequence of the actual input video variational autoencoder is: The extended video frame sequence contains 28 video frames.

[0525] In the implementation where the time dimension of the output latent variable of the video variational autoencoder is equal to the number of input video frames divided by the target temporal downsampling factor, the initial latent variable obtained in step S2083... It includes 7 latent variable time steps.

[0526] Among them, the left context video frame corresponds to 1 latent variable time step, the target time series segment interval corresponds to 5 latent variable time steps, and the right context video frame corresponds to 1 latent variable time step.

[0527] Therefore, in step S2084, the following can be retained from the initial latent variables: Among them, the latent variable start time index 1 is used to remove the context latent variables corresponding to the left context video frame; the latent variable end time index 6 adopts the right open interval form. Therefore, the latent variable time steps corresponding to time indices 1, 2, 3, 4 and 5 are actually retained, and the right context latent variables corresponding to latent variable time index 6 are removed.

[0528] Thus, the obtained It includes 5 latent variable time steps, corresponding to the 20 original video frames contained in the target time-series segmentation interval. This can be... The target time series segmentation interval was determined. The corresponding latent target variables.

[0529] In this way, when there are no reusable latent variables that meet the conditions for latent variable reuse, the target latent variable can be generated by a video variational autoencoder with context.

[0530] Compared to directly encoding the video frame sequence of the target temporal segmentation interval independently, this embodiment improves the continuity and representation accuracy of video latent variables in the temporal dimension by introducing contextual video frames that meet the effective temporal receptive field requirements of the video variational autoencoder.

[0531] Meanwhile, by removing the contextual latent variables corresponding to the contextual video frames, it can be ensured that the final target latent variables still maintain a correspondence with the video content range of the target temporal segment interval.

[0532] In some alternative implementations, step S209 may be included after step S208.

[0533] Step S209: Save the target latent variables and the corresponding video processing configuration, and use the target latent variables for training the vision-language-action model.

[0534] In step S209, after obtaining the target latent variable corresponding to the target time segment interval, the target latent variable and the video processing configuration corresponding to the target latent variable can be saved, and the target latent variable can be used for training the vision-language-action model.

[0535] In some embodiments of this application, the target latent variable is used to represent the low-dimensional feature representation of the video information corresponding to the robot task trajectory data after being encoded by a video variational autoencoder.

[0536] The target latent variable can correspond to the video content of the target time-series segment interval and is related to the video processing configuration used in the process of generating the target latent variable.

[0537] Specifically, the video processing configuration may include target motion configuration parameters and the target temporal downsampling factor corresponding to the video variational autoencoder.

[0538] Since different robot task trajectory data may correspond to different task temporal sensitivity levels and motion complexity levels, the video processing configuration used to generate target latent variables in different target temporal segment intervals may differ.

[0539] Therefore, after obtaining the target latent variables, the target latent variables and the corresponding video processing configuration can be saved simultaneously to establish the correspondence between the target latent variables and the video processing process.

[0540] For example, the target latent variables, the video time range corresponding to the target temporal segment intervals, the target action configuration parameters, and the target temporal downsampling factor can be stored together.

[0541] In the subsequent training of the vision-language-action model, the corresponding target latent variables can be obtained according to the training task requirements, and the video encoding method and time dimension processing method corresponding to the target latent variables can be determined according to the saved video processing configuration.

[0542] By saving the target latent variables and the corresponding video processing configuration, the video variational autoencoder encoding process can be repeatedly executed during the training of the vision-language-action model, thereby improving the efficiency of training data processing.

[0543] Meanwhile, since the target latent variables are associated with the corresponding video processing configuration, it can be ensured that the video latent variables corresponding to different robot task trajectory data have consistent temporal encoding relationships and task semantic correspondences during model training.

[0544] The above method can be used to generate and store latent variables corresponding to the video information of robot task trajectory data, providing time-optimized video latent variable training data for subsequent vision-language-action model training.

[0545] The video latent variable generation method provided in the embodiments of this application acquires robot task trajectory data including video frame sequences, robot motion sequences, and task attribute information. It determines the task temporal sensitivity level based on the task attribute information, determines the motion complexity level based on the video frame sequences and robot motion sequences, and then determines the video processing configuration based on the task temporal sensitivity level and motion complexity level. This allows the video processing process to be adjusted in conjunction with the task characteristics and motion change characteristics of the robot task data itself, thereby improving the adaptability of the video latent variable generation process to different robot task trajectory data.

[0546] Furthermore, by determining multiple target temporal segmentation intervals that satisfy the temporal coding constraints of the video variational autoencoder in the video frame sequence according to the video processing configuration, the number of video frames corresponding to the target temporal segmentation intervals can meet the temporal coding requirements of the video variational autoencoder, thereby improving the stability of the correspondence between the video temporal range and the temporal dimension of the video latent variables.

[0547] Furthermore, for any target temporal segmentation interval, the existence of reusable latent variables is determined based on the latent variable reuse condition. If a reusable latent variable that meets the latent variable reuse condition exists, the target latent variable is obtained based on the reusable latent variable. If no reusable latent variable that meets the latent variable reuse condition exists, a video variational autoencoder is executed to generate the target latent variable. This allows existing video latent variables to be effectively utilized when the reuse condition is met, reducing the repeated encoding process and improving the efficiency of video latent variable generation.

[0548] In summary, the technical solution provided in this application can determine the appropriate video processing configuration based on the characteristics of robot task trajectory data, generate video latent variables that meet the temporal coding constraints of the video variational autoencoder, and simultaneously enable the reuse of qualified video latent variables, thereby improving the effectiveness of the video latent variable generation process.

[0549] Further reference Figure 8 , Figure 8 A schematic diagram of an embodiment of the video latent variable generation apparatus according to this application is shown. As an implementation of the methods shown in the above figures, this application provides an embodiment of the video latent variable generation apparatus, which is similar to... Figure 2 Corresponding to the illustrated embodiment of the video latent variable generation method, this device can be specifically applied to various terminal devices or servers.

[0550] like Figure 8 As shown, the video latent variable generation device 800 may include a task trajectory acquisition module 801, a task temporal sensitivity level determination module 802, a motion complexity level determination module 803, a video processing configuration determination module 804, a temporal segmentation interval determination module 805, a latent variable reuse judgment module 806, a latent variable reuse module 807, and a latent variable encoding module 808.

[0551] The task trajectory acquisition module 801 is used to acquire robot task trajectory data, which includes video frame sequences, robot action sequences, and task attribute information.

[0552] The task timing sensitivity level determination module 802 is used to determine the task timing sensitivity level of the robot task trajectory data based on task attribute information.

[0553] The motion complexity level determination module 803 is used to determine the motion complexity level of the robot task trajectory data based on the video frame sequence and the robot motion sequence.

[0554] The video processing configuration determination module 804 is used to determine the video processing configuration based on the task's temporal sensitivity level and motion complexity level; wherein, the video processing configuration includes target motion configuration parameters and the target temporal downsampling factor corresponding to the video variational autoencoder.

[0555] The temporal segmentation interval determination module 805 is used to determine multiple target temporal segmentation intervals in the video frame sequence that satisfy the temporal coding constraints of the video variational autoencoder, based on the video processing configuration; wherein, the temporal coding constraints of the video variational autoencoder include the time alignment constraint between the number of input video frames and the target temporal downsampling factor.

[0556] The latent variable reuse judgment module 806 is used to determine whether there is a reusable latent variable corresponding to the target time series segment interval for any target time series segment interval based on the latent variable reuse condition, and to determine the target latent variable corresponding to the target time series segment interval based on the judgment result.

[0557] The latent variable reuse module 807 is used to obtain the target latent variable corresponding to the target time series segment interval based on the reusable latent variable when there is a reusable latent variable that meets the latent variable reuse condition.

[0558] The latent variable encoding module 808 is used to perform video variational autoencoder encoding on the video frame sequence of the target temporal segment interval when there is no reusable latent variable that meets the latent variable reuse condition, so as to obtain the target latent variable of the target temporal segment interval.

[0559] In some alternative implementations, the motion complexity level determination module 803 can also be used for: Based on the video content change information between adjacent video frames in the video frame sequence, video motion features of the video frame sequence are generated; among them, video motion features are used to characterize the overall video motion level, the degree of video motion change, and the peak video motion of the corresponding video frame sequence.

[0560] Based on the motion change information between adjacent robot actions in the robot motion sequence, motion motion features of the robot motion sequence are generated; among them, motion motion features are used to characterize the overall level of motion change, degree of motion change, and peak of motion change corresponding to the robot motion sequence.

[0561] Based on the motion characteristics of the video and the motion characteristics of the action, the motion complexity corresponding to the robot task trajectory data is determined.

[0562] Obtain the preset threshold for classifying motion complexity levels, and determine the motion complexity level of the robot task trajectory data based on the threshold.

[0563] In some optional implementations, the task attribute information includes at least one of robot state machine stage information, task tags, skill identifiers, and language instruction information. The task timing sensitivity level determination module 802 can also be used for: Obtain a pre-defined first correspondence table representing the relationship between a set of task time sensitivity levels and at least one of robot state machine stage information, task labels, and skill identifiers; wherein the set of task time sensitivity levels includes high time sensitivity level, medium time sensitivity level, and low time sensitivity level.

[0564] When the task attribute information contains at least one of robot state machine stage information, task tag, or skill identifier, the task timing sensitivity level corresponding to the robot task trajectory data is determined according to the first correspondence table.

[0565] When the task attribute information does not include robot state machine stage information, task tags, and skill identifiers, task feature recognition is performed based on language instruction information, and the task timing sensitivity level corresponding to the robot task trajectory data is determined based on the recognized task features.

[0566] In some alternative implementations, the video processing configuration determination module 804 may also be used for: Obtain a preset set of candidate action configuration parameters and a set of candidate time-series downsampling factors; wherein, the set of candidate action configuration parameters includes at least one candidate action configuration parameter, the candidate action configuration parameter includes the number of historical observation steps, the number of action prediction steps and the observation sampling span, and the set of candidate time-series downsampling factors includes at least one candidate time-series downsampling factor.

[0567] Based on the task's temporal sensitivity level and motion complexity level, the target temporal downsampling factor is determined from the candidate temporal downsampling factor set, and the target action configuration parameter is determined from the candidate action configuration parameter set.

[0568] In some alternative implementations, the video processing configuration determination module 804 may also be used for: Based on a pre-defined second correspondence table that characterizes the relationship between task temporal sensitivity level, motion complexity level and each candidate temporal downsampling factor, the target temporal downsampling factor is determined from the set of candidate temporal downsampling factors.

[0569] Based on a pre-defined third correspondence table representing the relationship between task temporal sensitivity level, motion complexity level, and configuration parameters of each candidate action, the target action configuration parameters are determined from the set of candidate action configuration parameters; or, Input the task timing sensitivity level and / or motion complexity level into the preset configuration selection model to obtain the selection probability corresponding to each candidate action configuration parameter; determine the candidate action configuration parameter with the highest selection probability based on the selection probability corresponding to each candidate action configuration parameter; and determine the candidate action configuration parameter with the highest selection probability as the target action configuration parameter.

[0570] In some optional implementations, the timing segmentation interval determination module 805 can also be used for: The temporal segmentation interval length of the video frame sequence is determined based on the target action configuration parameters and the target temporal downsampling factor.

[0571] The video frame sequence is divided according to the length of the temporal segment interval to obtain multiple initial temporal segment intervals.

[0572] Determine whether there are any initial temporal segmentation intervals among multiple initial temporal segmentation intervals that do not satisfy the temporal coding constraints of the video variational autoencoder.

[0573] If none exist, multiple initial temporal segmentation intervals are determined as multiple target temporal segmentation intervals that satisfy the temporal coding constraints of the video variational autoencoder.

[0574] If it exists, adjust the initial temporal segment interval that does not meet the temporal coding constraints of the video variational autoencoder according to the length of the temporal segment interval to obtain the adjusted initial temporal segment interval, wherein the adjusted initial temporal segment interval meets the temporal coding constraints of the video variational autoencoder.

[0575] The initial temporal segmentation interval that satisfies the temporal coding constraints of the video variational autoencoder, as well as the adjusted initial temporal segmentation interval, are determined as multiple target temporal segmentation intervals that satisfy the temporal coding constraints of the video variational autoencoder.

[0576] In some optional implementations, the timing segmentation interval determination module 805 can also be used for: The number of supplementary video frames to be added is determined based on the length of the time-series segment interval.

[0577] Supplementary video frames are generated based on the number of supplementary video frames.

[0578] The supplementary video frames are added to the initial temporal segmentation interval that does not meet the temporal coding constraints of the video variational autoencoder, to obtain the adjusted initial temporal segmentation interval, which includes the original video frames and the supplementary video frames.

[0579] In some alternative implementations, the latent variable reuse determination module 806 can also be used for: Query at least one existing valid latent variable that has been encoded by a variational autoencoder for video.

[0580] Determine the parent time series segment intervals for each existing effective latent variable.

[0581] Determine whether there exists a reusable parent time series segment interval that satisfies the latent variable reuse condition.

[0582] When a reusable parent time series segment interval exists, it is determined that there is a reusable latent variable corresponding to the target time series segment interval.

[0583] When there is no reusable parent time series segment interval, it is determined that there is no reusable latent variable corresponding to the target time series segment interval.

[0584] Latent variable reuse conditions include at least one of the following: time interval coverage condition, time base consistency condition, time index mapping condition, time-series coding condition, context sufficiency condition, and coding condition consistency condition.

[0585] The time interval coverage condition is used to determine whether the parent time series segment interval covers the target time series segment interval.

[0586] The time reference consistency condition is used to determine whether the parent time segment interval and the target time segment interval are determined based on the same original video frame time axis.

[0587] The time index mapping condition is used to determine whether the offset of the starting position of the target time series segment interval relative to the parent time series segment interval meets the time index mapping requirements of the target time series downsampling factor.

[0588] The timing coding condition is used to determine whether the parent timing segment interval and the target timing segment interval use the same timing downsampling factor of the video variational autoencoder.

[0589] The context sufficiency condition is used to determine whether the parent temporal segment interval can provide context information that meets the temporal coding requirements of the video variational autoencoder for the target temporal segment interval.

[0590] The coding condition consistency condition is used to determine whether the coding conditions of the video variational autoencoder corresponding to the parent temporal segment interval and the target temporal segment interval are consistent.

[0591] In some alternative implementations, the latent variable reuse module 807 can also be used for: Based on the time offset of the target time series segment interval relative to the reusable parent time series segment interval, the reusable latent variable time index range corresponding to the reusable latent variable is determined.

[0592] Based on the time index range of reusable latent variables, time slicing is performed on the reusable latent variables to obtain the target latent variables for the target time series segment interval.

[0593] In some alternative implementations, the latent variable encoding module 808 can also be used for: Obtain the effective temporal receptive field range of the video variational autoencoder; Based on the effective temporal receptive field range, context video frames are added to the left and / or right of the target temporal segmentation interval to generate an extended video frame sequence, and the context video frame index range corresponding to the context video frames in the extended video frame sequence is determined.

[0594] Perform video variational autoencoder encoding on the extended video frame sequence to obtain the initial latent variables of the target temporal segmentation interval.

[0595] Based on the context video frame index range and the target temporal downsampling factor, determine the context latent variables corresponding to the context video frames in the initial latent variables, and remove the context latent variables.

[0596] The remaining latent variables after removing contextual latent variables are determined as the target latent variables for the target time series segment interval.

[0597] In some alternative implementations, the video latent variable generation apparatus 800 may further include a boundary adjustment module 809. The boundary adjustment module 809 is used for: For any target time series segmentation interval, within a preset boundary range, determine multiple candidate boundary adjustment positions.

[0598] Based on the video frame timestamps corresponding to the adjustment positions of each candidate boundary and the action timestamps corresponding to each robot action in the robot action sequence, the target action index corresponding to each candidate boundary adjustment position is determined.

[0599] Based on the target action index corresponding to the position of each candidate boundary adjustment, determine the robot action data corresponding to each target action index.

[0600] Based on the robot's motion data, the first boundary evaluation result corresponding to the adjustment position of each candidate boundary is determined.

[0601] The candidate boundary adjustment position with the smallest first boundary evaluation result is determined as the target boundary adjustment position.

[0602] The boundary translation amount is determined based on the position difference between the target boundary adjustment position and the current boundary position of the target temporal segment interval. The boundary translation amount is used to represent the translation direction and translation distance of the target temporal segment interval in the video frame sequence.

[0603] Based on the boundary translation amount, the starting and ending boundaries of the target time series segment interval are synchronously translated in the same direction and by the same distance to obtain the first target time series segment interval after translation.

[0604] In some optional implementations, when multiple target temporal segment intervals are used as independent training samples, the video latent variable generation device 800 may further include an independent sample boundary optimization module 810. The independent sample boundary optimization module 810 is used for: For any target time series segmentation interval, multiple legal candidate boundary shifts are determined; where each legal candidate boundary shift is an integer multiple of the target time series downsampling factor.

[0605] Based on multiple legitimate candidate boundary translations, the starting and ending boundaries of the target time series segment intervals are synchronously translated to obtain multiple candidate time series segment intervals that correspond one-to-one with each legitimate candidate boundary translation.

[0606] Determine the second boundary evaluation results for each candidate time series segment interval.

[0607] The candidate time series segment interval with the smallest second boundary evaluation result is determined as the second target time series segment interval after translation.

[0608] In some optional implementations, when a continuous temporal segment sequence consisting of multiple target temporal segment intervals exists within multiple target temporal segment intervals, the video latent variable generation device 800 may further include a continuous segment global optimization module 811. The continuous segment global optimization module 811 can be used for: Determine the range of video frame indices corresponding to the continuous temporal segment sequence.

[0609] Multiple candidate global segment boundary shifts are determined based on the segmented sequence video frame index range.

[0610] Based on the preset legal constraints of global segment boundary translation, multiple candidate global segment boundary translations are screened to obtain legal candidate global segment boundary translations.

[0611] Based on the translation amount of each legal candidate global segment boundary, the segment boundary corresponding to the continuous time series segment sequence is translated to obtain the corresponding candidate continuous time series segment sequence.

[0612] Determine the global segmentation boundary evaluation results for each candidate continuous time series segmentation sequence.

[0613] The candidate continuous temporal segment sequence with the smallest global segment boundary evaluation result is determined as the translated continuous temporal segment sequence.

[0614] In some optional embodiments, the video latent variable generation apparatus 800 may further include a multi-granularity latent variable multiplexing module 812. The multi-granularity latent variable multiplexing module 812 can be used for: For the same robot task trajectory data, coarse-grained time-series segmentation intervals and fine-grained time-series segmentation intervals are established according to different time granularities. The range of coarse-grained time-series segmented video frame indexes corresponding to the coarse-grained time-series segmentation intervals is larger than the range of fine-grained time-series segmented video frames corresponding to the fine-grained time-series segmentation intervals.

[0615] When a fine-grained time series segmentation interval is completely covered by a single coarse-grained time series segmentation interval, the fine-grained time series segmentation latent variable corresponding to the fine-grained time series segmentation interval is determined based on the coarse-grained time series segmentation latent variable corresponding to the coarse-grained time series segmentation interval.

[0616] When a fine-grained temporal segmentation interval crosses the boundary position between multiple coarse-grained temporal segmentation intervals, video variational autoencoder encoding is performed on the fine-grained temporal segmentation interval to obtain the fine-grained temporal segmentation latent variables corresponding to the fine-grained temporal segmentation interval.

[0617] In some alternative implementations, the video latent variable generation device 800 may further include a model training application module 813. The model training application module 813 is used for: After obtaining the target latent variables, save the target latent variables and the corresponding video processing configuration, and use the target latent variables for training the vision-language-action model.

[0618] It should be understood that each module described in device 800 corresponds to the corresponding step in the foregoing method embodiments. Therefore, the operations, features, and technical effects described for each step in the foregoing method embodiments are also applicable to the corresponding modules in device 800, and will not be repeated here.

[0619] It should also be understood that Figure 8 The modules shown are for illustrative purposes only and do not constitute a limitation on the device structure. In actual implementation, the above modules can be combined into one module, and a single module can be further divided into multiple sub-modules. The above modules can be implemented using software, hardware, or a combination of both.

[0620] It should be noted that the implementation details and technical effects of each module in the video latent variable generation device provided in the embodiments of this application can be referred to the descriptions of other embodiments in this application, and will not be repeated here.

[0621] like Figure 9As shown, the computer system 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the computer system 900. The processing device 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0622] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows computer system 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 A computer system 900 with various electronic devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0623] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this application.

[0624] It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0625] In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0626] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), or any suitable combination thereof.

[0627] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0628] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the following functions: Figure 2 The steps shown in the embodiment of the video latent variable generation method and its optional implementation are illustrated.

[0629] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0630] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0631] The units described in the embodiments of this application can be implemented in software or hardware. The names of the units are not necessarily limiting in certain circumstances; for example, a task trajectory acquisition module can also be described as "a module for acquiring robot task trajectory data".

[0632] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with, but not limited to, technical features disclosed in this application that have similar functions.

Claims

1. A method for generating latent variables in video, characterized in that, The method includes: Acquire robot task trajectory data, which includes video frame sequences, robot action sequences, and task attribute information; The task timing sensitivity level of the robot task trajectory data is determined based on the task attribute information. The motion complexity level of the robot task trajectory data is determined based on the video frame sequence and the robot motion sequence. The video processing configuration is determined based on the task temporal sensitivity level and the motion complexity level; wherein, the video processing configuration includes target motion configuration parameters and the target temporal downsampling factor corresponding to the video variational autoencoder; According to the video processing configuration, multiple target temporal segmentation intervals that satisfy the temporal coding constraints of the video variational autoencoder are determined in the video frame sequence; wherein, the temporal coding constraints of the video variational autoencoder include the temporal alignment constraint between the number of input video frames and the target temporal downsampling factor; For any of the target time series segment intervals, determine whether there are reusable latent variables corresponding to the target time series segment intervals based on the latent variable reuse conditions, and determine the target latent variables corresponding to the target time series segment intervals based on the determination results; When there is a reusable latent variable that satisfies the latent variable reuse condition, the target latent variable for the target time series segment interval is obtained based on the reusable latent variable; When there is no reusable latent variable that satisfies the latent variable reuse condition, video variational autoencoder encoding is performed on the video frame sequence of the target temporal segment interval to obtain the target latent variable of the target temporal segment interval.

2. The method according to claim 1, characterized in that, The step of determining the motion complexity level corresponding to the robot task trajectory data based on the video frame sequence and the robot motion sequence includes: Based on the video content change information between adjacent video frames in the video frame sequence, video motion features of the video frame sequence are generated; wherein, the video motion features are used to characterize the overall video motion level, the degree of video motion change, and the peak video motion of the video frame sequence. Based on the motion change information between adjacent robot actions in the robot motion sequence, motion motion features of the robot motion sequence are generated; wherein, the motion motion features are used to characterize the overall motion change level, motion change degree, and motion change peak of the robot motion sequence. Based on the video motion features and the action motion features, determine the motion complexity corresponding to the robot task trajectory data; Obtain a preset threshold for classifying motion complexity levels, and determine the motion complexity level of the robot task trajectory data based on the threshold for classifying motion complexity levels.

3. The video latent variable generation method according to claim 1, characterized in that, The task attribute information includes at least one of robot state machine stage information, task tags, skill identifiers, and language instruction information. Determining the task temporal sensitivity level of the robot task trajectory data based on the task attribute information includes: Obtain a preset first correspondence table representing the relationship between a set of task time sensitivity levels and at least one of robot state machine stage information, task tags, and skill identifiers; wherein the set of task time sensitivity levels includes high time sensitivity level, medium time sensitivity level, and low time sensitivity level; When the task attribute information includes at least one of robot state machine stage information, task tag or skill identifier, the task time sensitivity level corresponding to the robot task trajectory data is determined according to the first correspondence table. When the task attribute information does not include robot state machine stage information, task tags, and skill identifiers, task feature recognition is performed based on the language instruction information, and the task timing sensitivity level corresponding to the robot task trajectory data is determined based on the recognized task features.

4. The method according to claim 1, characterized in that, The step of determining the video processing configuration based on the task's temporal sensitivity level and the motion complexity level includes: Obtain a preset set of candidate action configuration parameters and a set of candidate time-series downsampling factors; wherein, the set of candidate action configuration parameters includes at least one candidate action configuration parameter, the candidate action configuration parameter includes the number of historical observation steps, the number of action prediction steps, and the observation sampling span, and the set of candidate time-series downsampling factors includes at least one candidate time-series downsampling factor; Based on the task timing sensitivity level and the motion complexity level, the target timing downsampling factor is determined from the candidate timing downsampling factor set, and the target action configuration parameter is determined from the candidate action configuration parameter set.

5. The method according to claim 4, characterized in that, The step of determining the target temporal downsampling factor from the candidate temporal downsampling factor set based on the task temporal sensitivity level and the motion complexity level, and determining the target action configuration parameter from the candidate action configuration parameter set, includes: The target temporal downsampling factor is determined from the set of candidate temporal downsampling factors according to a pre-set second correspondence table used to characterize the correspondence between the task temporal sensitivity level, the motion complexity level and each candidate temporal downsampling factor; Based on a pre-defined third correspondence table characterizing the relationship between the task's temporal sensitivity level, the motion complexity level, and each candidate action configuration parameter, the target action configuration parameter is determined from the set of candidate action configuration parameters; or, Input the task timing sensitivity level and / or the motion complexity level into a preset configuration selection model to obtain the selection probability corresponding to each candidate action configuration parameter; determine the candidate action configuration parameter with the highest selection probability based on the selection probability corresponding to each candidate action configuration parameter; and determine the candidate action configuration parameter with the highest selection probability as the target action configuration parameter.

6. The method according to claim 1, characterized in that, The step of determining multiple target temporal segmentation intervals in the video frame sequence that satisfy the temporal coding constraints of the video variational autoencoder, according to the video processing configuration, includes: The temporal segmentation interval length of the video frame sequence is determined based on the target action configuration parameters and the target temporal downsampling factor; The video frame sequence is divided according to the length of the time segmentation interval to obtain multiple initial time segmentation intervals; Determine whether there are any initial temporal segmentation intervals among the plurality of initial temporal segmentation intervals that do not satisfy the temporal coding constraints of the video variational autoencoder; If not, the plurality of initial temporal segmentation intervals are determined as the plurality of target temporal segmentation intervals that satisfy the temporal coding constraints of the video variational autoencoder; If it exists, the initial temporal segment interval that does not meet the temporal coding constraints of the video variational autoencoder is adjusted according to the temporal segment interval length to obtain the adjusted initial temporal segment interval, wherein the adjusted initial temporal segment interval meets the temporal coding constraints of the video variational autoencoder. The initial temporal segmentation interval that satisfies the temporal coding constraints of the video variational autoencoder and the adjusted initial temporal segmentation interval are determined as the plurality of target temporal segmentation intervals that satisfy the temporal coding constraints of the video variational autoencoder.

7. The method according to claim 6, characterized in that, The step of adjusting the initial temporal segment intervals that do not meet the temporal coding constraints of the video variational autoencoder according to the length of the temporal segment interval to obtain the adjusted initial temporal segment intervals includes: The number of supplementary video frames to be added is determined based on the length of the time segment interval. Generate supplementary video frames based on the number of supplementary video frames; The supplementary video frame is added to the initial temporal segmentation interval that does not meet the temporal coding constraints of the video variational autoencoder to obtain the adjusted initial temporal segmentation interval, wherein the adjusted initial temporal segmentation interval includes the original video frame and the supplementary video frame.

8. The method according to claim 1, characterized in that, The step of determining whether there are reusable latent variables based on the latent variable reuse condition includes: Query at least one existing valid latent variable that has been encoded by a variational autoencoder for video; Determine the parent time series segment interval for each of the existing effective latent variables; Determine whether there exists a reusable parent time series segment interval that satisfies the latent variable reuse condition; When the reusable parent time series segmentation interval exists, it is determined that there is a reusable latent variable corresponding to the target time series segmentation interval; When the reusable parent time series segmentation interval does not exist, it is determined that there is no reusable latent variable corresponding to the target time series segmentation interval.

9. The method according to claim 8, characterized in that, The latent variable reuse conditions include at least one of the following: time interval coverage condition, time base consistency condition, time index mapping condition, time-series coding condition, context sufficiency condition, and coding condition consistency condition; wherein... The time interval coverage condition is used to determine whether the parent time segment interval covers the target time segment interval; The time reference consistency condition is used to determine whether the parent time segment interval and the target time segment interval are determined based on the same original video frame time axis. The time index mapping condition is used to determine whether the offset of the starting position of the target time segment interval relative to the parent time segment interval meets the time index mapping requirement of the target time downsampling factor; The timing coding conditions are used to determine whether the parent timing segmentation interval and the target timing segmentation interval use the same video variational autoencoder timing downsampling factor; The context sufficiency condition is used to determine whether the parent temporal segmentation interval can provide context information that satisfies the temporal coding requirements of the video variational autoencoder for the target temporal segmentation interval; The coding condition consistency condition is used to determine whether the coding conditions of the video variational autoencoder corresponding to the parent temporal segment interval and the target temporal segment interval are consistent.

10. The method according to claim 8, characterized in that, The process of obtaining the target latent variable for the target time series segment interval based on the reusable latent variable includes: Based on the time offset of the target time series segment interval relative to the reusable parent time series segment interval, determine the reusable latent variable time index range corresponding to the reusable latent variable; Based on the reusable latent variable time index range, time slicing is performed on the reusable latent variable to obtain the target latent variable in the target time series segment interval.

11. The method according to claim 1, characterized in that, The step of performing video variational autoencoder encoding on the video frame sequence of the target temporal segmentation interval to obtain the target latent variable of the target temporal segmentation interval includes: Obtain the effective temporal receptive field range of the video variational autoencoder; Based on the effective temporal receptive field range, context video frames are added to the left and / or right of the target temporal segmentation interval to generate an extended video frame sequence, and the context video frame index range corresponding to the context video frames in the extended video frame sequence is determined. The extended video frame sequence is encoded using a video variational autoencoder to obtain the initial latent variables of the target temporal segmentation interval; Based on the context video frame index range and the target temporal downsampling factor, determine the context latent variable corresponding to the context video frame in the initial latent variable, and remove the context latent variable; The remaining latent variables after removing the context latent variables are determined as the target latent variables of the target time series segment interval.

12. The method according to claim 1, characterized in that, After determining, according to the video processing configuration, multiple target temporal segmentation intervals in the video frame sequence that satisfy the temporal coding constraints of the video variational autoencoder, the method further includes: For any of the target time-series segmentation intervals, within a preset boundary range, multiple candidate boundary adjustment positions are determined; Based on the video frame timestamps corresponding to each candidate boundary adjustment position and the action timestamps corresponding to each robot action in the robot action sequence, determine the target action index corresponding to each candidate boundary adjustment position; based on the target action index corresponding to each candidate boundary adjustment position, determine the robot action data corresponding to each target action index; Based on the robot motion data, determine the first boundary evaluation result corresponding to each candidate boundary adjustment position; The candidate boundary adjustment position with the smallest first boundary evaluation result is determined as the target boundary adjustment position; Based on the positional difference between the target boundary adjustment position and the current boundary position of the target temporal segment interval, the boundary translation amount is determined. The boundary translation amount is used to represent the translation direction and translation distance of the target temporal segment interval in the video frame sequence. According to the boundary translation amount, the starting boundary and ending boundary of the target temporal segment interval are synchronously translated in the same direction and by the same distance to obtain the first target temporal segment interval after translation.

13. The method according to claim 1, characterized in that, When multiple target temporal segmentation intervals are used as independent training samples, after determining multiple target temporal segmentation intervals that satisfy the temporal coding constraints of the video variational autoencoder in the video frame sequence according to the video processing configuration, the method further includes: For any of the target time series segmentation intervals, multiple valid candidate boundary shifts are determined; wherein each valid candidate boundary shift is an integer multiple of the target time series downsampling factor; Based on multiple legitimate candidate boundary translations, the starting and ending boundaries of the target time series segment intervals are synchronously translated to obtain multiple candidate time series segment intervals that correspond one-to-one with each legitimate candidate boundary translation. Determine the second boundary evaluation results for each candidate time series segment interval; The candidate time series segment interval with the smallest second boundary evaluation result is determined as the second target time series segment interval after translation.

14. The method according to claim 1, characterized in that, When a continuous temporal segment sequence consisting of multiple target temporal segment intervals exists among the multiple target temporal segment intervals, after determining multiple target temporal segment intervals satisfying the temporal coding constraints of the video variational autoencoder in the video frame sequence according to the video processing configuration, the method further includes: Determine the range of video frame indices for the segmented sequence corresponding to the continuous temporal segmented sequence; Multiple candidate global segment boundary translation amounts are determined based on the segmented sequence video frame index range; Based on the preset legal constraints of global segment boundary translation, the multiple candidate global segment boundary translations are filtered to obtain legal candidate global segment boundary translations. Based on the translation amount of each legal candidate global segment boundary, the segment boundary corresponding to the continuous temporal segment sequence is translated to obtain the corresponding candidate continuous temporal segment sequence. Determine the global segmentation boundary evaluation results for each of the candidate continuous temporal segmentation sequences; The candidate continuous temporal segment sequence with the smallest global segment boundary evaluation result is determined as the translated continuous temporal segment sequence.

15. The method according to claim 8, characterized in that, The method further includes: For the same robot task trajectory data, coarse-grained time-series segmentation intervals and fine-grained time-series segmentation intervals are established according to different time granularities. The coarse-grained time-series segmentation video frame index range corresponding to the coarse-grained time-series segmentation interval is larger than the fine-grained time-series segmentation video frame range corresponding to the fine-grained time-series segmentation interval. When the fine-grained time series segmentation interval is completely covered by a single coarse-grained time series segmentation interval, the fine-grained time series segmentation latent variable corresponding to the fine-grained time series segmentation interval is determined according to the coarse-grained time series segmentation latent variable corresponding to the coarse-grained time series segmentation interval. When the fine-grained temporal segmentation interval crosses the boundary position between multiple coarse-grained temporal segmentation intervals, video variational autoencoder encoding is performed on the fine-grained temporal segmentation interval to obtain the fine-grained temporal segmentation latent variable corresponding to the fine-grained temporal segmentation interval.

16. The method according to claim 1, characterized in that, After obtaining the target latent variable, the method further includes: Save the target latent variables and the corresponding video processing configuration, and use the target latent variables for training the vision-language-action model.

17. A video latent variable generation device, characterized in that, The device includes: The task trajectory acquisition module is used to acquire robot task trajectory data, which includes video frame sequences, robot action sequences, and task attribute information. The task timing sensitivity level determination module is used to determine the task timing sensitivity level of the robot task trajectory data based on the task attribute information. A motion complexity level determination module is used to determine the motion complexity level of the robot task trajectory data based on the video frame sequence and the robot action sequence; A video processing configuration determination module is used to determine a video processing configuration based on the task temporal sensitivity level and the motion complexity level; wherein, the video processing configuration includes target motion configuration parameters and a target temporal downsampling factor corresponding to the video variational autoencoder; The temporal segmentation interval determination module is used to determine, according to the video processing configuration, multiple target temporal segmentation intervals in the video frame sequence that satisfy the temporal coding constraints of the video variational autoencoder; wherein, the temporal coding constraints of the video variational autoencoder include a time alignment constraint between the number of input video frames and the target temporal downsampling factor; The latent variable reuse judgment module is used to determine, for any target time series segment interval, whether there is a reusable latent variable corresponding to the target time series segment interval according to the latent variable reuse condition, and to determine the target latent variable corresponding to the target time series segment interval according to the judgment result; The latent variable reuse module is used to obtain the target latent variable of the target time series segment interval based on the reusable latent variable when there is a reusable latent variable that satisfies the latent variable reuse condition; The latent variable encoding module is used to perform video variational autoencoder encoding on the video frame sequence of the target temporal segmentation interval when there is no reusable latent variable that satisfies the latent variable reuse condition, so as to obtain the target latent variable of the target temporal segmentation interval.

18. An electronic device comprising: One or more processors; Storage device, on which one or more programs are stored, When one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1-16.

19. A computer-readable storage medium having a computer program stored thereon, wherein, When a computer program is executed by one or more processors, it implements the method as described in any one of claims 1-16.

20. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method as described in any one of claims 1-16.

Images