Robot long-term operation control model training method, control method and system

Abstract

The application provides a robot long-time operation control model training method, a control method and a system, and relates to the technical field of data processing. The training method comprises the following steps: encoding language instructions, observation images and trajectory data; using text encoding, image encoding and trajectory encoding to construct a prediction input sequence, inputting the prediction input sequence into a trajectory prediction model, obtaining first prediction data, comparing the first prediction data with first actual data, and comparing until a total cross-entropy loss value meets a requirement, so as to obtain the trajectory prediction model; extracting features from the language instructions, the observation images, segmented images and actual trajectory sequences, using text features, multi-modal features, actual trajectory features and task states to construct strategy input features, inputting the strategy input features into a strategy generation model, obtaining second prediction data, comparing the second prediction data with second actual data, and comparing until a total loss value meets a requirement, so as to obtain the strategy generation model. A layered, robust and adaptive control model is constructed, error accumulation is reduced, and generalization capability is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a method, control method, and system for training a robot long-term operation control model. Background Technology

[0002] Robot end-effector control refers to the physical interaction between a robot and its environment through its end effectors (such as robotic arms, grippers, tools, etc.) to perform tasks such as grasping, assembly, polishing, and inspection. It has a wide range of applications, including industrial polishing, precision assembly, medical surgery, and underwater operations.

[0003] Many existing visual-language manipulation methods tend to build end-to-end models, directly mapping visual perception and language commands to underlying actions. This tight coupling can easily lead to the amplification of perceptual errors at each level during policy execution, especially in long-term tasks, where small initial deviations can cause subsequent actions to fail completely. Furthermore, this coupling limits the model's ability to generalize to unseen objects, scenes, or combinations of commands. Summary of the Invention

[0004] The problem that this invention aims to solve is that existing methods for constructing end-to-end models are prone to error accumulation and amplification, and have poor adaptability to the environment.

[0005] To address the aforementioned problems, in a first aspect, the present invention provides a method for training a long-term operation control model for a robot, comprising:

[0006] The language instructions and observed images are encoded to obtain text encoding and image encoding;

[0007] The trajectory data is subjected to discrete cosine transform, filtered, and compressed to obtain the trajectory code;

[0008] The prediction input sequence is constructed based on text encoding, image encoding and trajectory encoding, and then input into the trajectory prediction model to obtain the first prediction data for the next time period. The control model includes the trajectory prediction model and the policy generation model.

[0009] Based on the first predicted data and the first actual data, analyze the total cross-entropy loss value of the trajectory prediction model until the total cross-entropy loss value meets the requirements, and then stop training the trajectory prediction model.

[0010] Multimodal features are obtained by extracting features from language instructions, observed images, and segmented images. The segmented image is a mask image of the target object based on the observed image.

[0011] The actual trajectory sequence is input into the resampler to obtain the actual trajectory features;

[0012] Based on multimodal features, actual trajectory features, and task status, policy input features are constructed and input into the policy generation model to obtain the second prediction data for the next time period.

[0013] Based on the second predicted data and the second actual data, analyze the total loss value until the total loss value meets the requirements, then stop training the policy generation model.

[0014] Secondly, the present invention also provides a long-term operation control method for a robot, comprising:

[0015] The obtained language instructions, observation images, and initial position information of the execution end are encoded to obtain the predicted input sequence;

[0016] The predicted input sequence is input into the trajectory prediction model within the control model obtained by the above-mentioned robot long-term operation control model training method to obtain a predicted trajectory sequence of a preset length.

[0017] Features are extracted from language instructions, observed images, segmented images, and predicted trajectory sequences of a preset length to obtain policy input features;

[0018] Determine whether the gripper state remains consistent within the last specified number of steps in the predicted trajectory sequence of the preset length; if it remains consistent, maintain the weight of the trajectory guidance branch in the policy input features unchanged; otherwise, reset the weight of the trajectory guidance branch in the policy input features to zero.

[0019] The strategy input features are input into the strategy generation model within the control model obtained by the above-mentioned robot long-term operation control model training method to obtain the predicted action parameters of the execution end. Based on the predicted task state, the predicted action parameters are used to control the robot's execution end operation.

[0020] Thirdly, the present invention also provides a robot long-term operation control system, comprising:

[0021] The encoding module is used to encode the obtained language instructions, observation images, and initial position information of the execution end to obtain the predicted input sequence.

[0022] The trajectory prediction module is used to input the predicted input sequence into the trajectory prediction model within the control model obtained by the above-mentioned robot long-term operation control model training method, and obtain a predicted trajectory sequence of a preset length.

[0023] The feature extraction module is used to extract features from language instructions, observed images, segmented images, and predicted trajectory sequences of a preset length to obtain policy input features;

[0024] The weight switching module is used to determine whether the gripper state remains consistent within the last specified number of steps in the predicted trajectory sequence of a preset length. If it remains consistent, the weight of the trajectory guidance branch in the policy input feature is kept unchanged; otherwise, the weight of the trajectory guidance branch in the policy input feature is reset to zero.

[0025] The strategy generation module is used to input the strategy input features into the strategy generation model within the control model obtained by the above-mentioned robot long-term operation control model training method, to obtain the predicted action parameters of the execution end, and to control the robot's execution end to work according to the predicted task state.

[0026] This invention provides a method for training a robot long-term operation control model, a control method, and a system. Compared with existing technologies, it has the following advantages:

[0027] The strategy generation model uses actual trajectory data instead of the predicted trajectory sequence output by the trajectory prediction model. Training the two models separately in stages avoids error accumulation during training, improves training accuracy, and reduces the training load and data volume of the trajectory prediction model, thus enabling rapid development of the control model. In practical use, the trajectory prediction model in the control model outputs a predicted trajectory sequence of preset length. Based on this predicted trajectory sequence, the strategy generation model outputs predicted action parameters for the execution end, thereby controlling the movement of the execution end. By designing an independent trajectory prediction model, visual perception and understanding are effectively separated from the generation of action strategies at the execution end, i.e., perception and execution are decoupled. This trajectory prediction model is specifically responsible for parsing language commands and visual observation images, predicting particle trajectory sequences and gripper states, i.e., the future motion intentions of the execution end. Since each output is only a predicted trajectory sequence of a preset length, even if environmental interference or prediction errors occur during control operation, real-time observation images are obtained before the next prediction. The real-time observation images have a certain ability to correct errors and environmental interference during the prediction process, giving the trajectory prediction model self-correcting capabilities. Moreover, the predicted motion trajectory sequence is not directly used to control the execution end, but is used by the strategy generation model to generate the control parameters of the execution end. A hierarchical, robust, and adaptive control model is constructed. This decoupled architecture aims to significantly reduce the propagation and accumulation of perception errors in long action execution chains, making the system more stable when facing complex, multi-step, long-term manipulation tasks, and better generalizing to unseen new scenes, new objects, and novel command combinations. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart illustrating a method for training a robot long-term operation control model according to an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of trajectory data processing provided in an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of a control model architecture provided in an embodiment of the present invention;

[0032] Figure 4 A flowchart illustrating a long-term operation control method for a robot provided in an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram of a robot long-term operation control system provided in an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application are described clearly and completely. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0036] To train a long-term operation control model for a robot, data is first collected to prepare the training dataset. This dataset includes multiple data elements, each containing language commands, observed images, and a trajectory sequence (trajectory data) with a fixed number of steps. For example, within a specified time length, a fixed number of trajectory points are uniformly selected on a continuous motion trajectory to form the trajectory sequence. The observed images are those acquired by the global camera and the end effector camera at the starting point of that specified time length. The real-time movements of the robotic arm can be identified from the images acquired by the global camera, and the actual movements and gripper states at the end effector can be further precisely identified from the images acquired by the end effector. The entire robot motion time period can be divided into multiple motion time segments according to a specified time length. A data element can be extracted from each motion time segment, thus constructing the training dataset.

[0037] Since the position information of the execution end is a three-dimensional coordinate in the world coordinate system, it is necessary to project the three-dimensional coordinate information of the execution end onto the image plane. The conversion process is as follows.

[0038] First, determine the three-dimensional position of the end effector in the world coordinate system. Pixel position projected onto the image plane Assume the camera intrinsic parameter matrix of the global camera is as follows:

[0039] ;

[0040] in, , These represent the focal lengths in the horizontal and vertical directions, respectively. , These represent the x and y coordinates of the principal point, respectively.

[0041] The extrinsic parameters of the camera relative to the world coordinate system are represented as a transformation matrix that includes rotation and translation:

[0042] ,

[0043] in For rotation matrix, It is a translation vector.

[0044] By transforming the world coordinates to the camera coordinate system using extrinsic parameters, the three-dimensional position of the end effector in the camera coordinate system can be obtained. :

[0045] ;

[0046] in, , and These represent the three-dimensional coordinates of the end effector in the camera coordinate system;

[0047] Then, using camera intrinsic parameters, the points in the camera coordinate system are projected onto the image plane pixel coordinate system to obtain the pixel positions:

[0048] ;

[0049] During the data annotation phase, the aforementioned formula is used to accurately project the 3D trajectory of the robotic arm's end effector into pixel space to construct realistic trajectory labels. To further improve the accuracy of data annotation, existing video tracking technologies, such as KLT optical flow tracing and Siamese network-based tracking methods, can be combined to automatically extract pixel-level motion trajectories of the end effector (i.e., the execution end) from demonstration videos. These tracking technologies can supplement manual annotation and extract more diverse motion data.

[0050] For a trajectory sequence, for example, it can take the form of an actual trajectory sequence of length 10:

[0051] ,in, ,

[0052] in, Indicates the first The position vector at the end of the execution of a time step. and These represent the two-dimensional coordinates of the execution endpoint in the image coordinate system, with the top-left corner of the image as the origin. The depth values ​​from the depth map are normalized and used to construct three-dimensional spatial location information at a uniform scale. This indicates the gripper state; -1 indicates closed, and 1 indicates open.

[0053] Therefore, the data elements in the dataset are in the following form:

[0054] .

[0055] For training, before the start of the motion, there is no motion trajectory. At this time, only the initial position information can be input into the trajectory prediction model. Based on the initial position information, language commands, and observed images, the model outputs a predicted trajectory sequence. Then, based on the data elements corresponding to the first motion time segment, the model predicts the data corresponding to the second motion time segment, and so on, continuously training. The specific training process is as follows.

[0056] like Figure 1 As shown in the embodiment of this application, a method for training a robot long-term operation control model includes:

[0057] S1: Encode the language instructions and the observed images to obtain text codes and image codes. Each data in the dataset includes language instructions, observed images, trajectory data, and actual trajectory sequences.

[0058] Specifically, data input is provided by language instructions. Visual observation at the initial moment These components work together to provide the model with a comprehensive description of the task intent and environmental state. Language instructions can be things like "move the object to the right" or "pick up the red cube," and these instructions are then converted into text, such as... Figure 3 As shown, in the first stage, text and images are discretely encoded. For example, Figure 3 As shown, the observed images can include global images and end-effector images, acquired by the global camera and the end-effector camera respectively. The acquired images are saved sequentially in chronological order, forming a global camera sequence and an end-effector camera sequence. The global image displays the state of the entire robotic arm and its end effector, while the end-effector image primarily displays the state of the end effector, such as... Figure 2 As shown, the trajectory sequence of the execution end can be obtained from the global camera sequence.

[0059] S2: Perform discrete cosine transform, filter, and compress the trajectory data to obtain the trajectory code, where the observed image and the trajectory segment correspond in the time dimension.

[0060] Specifically, since the trajectory of the execution end is continuous, the actual trajectory data is continuous. However, for the control of the execution end, complete continuous trajectory data is not required. Only some stage-specific target points need to be extracted from the trajectory data for the control of the execution end. This avoids data redundancy and increases the anti-interference capability of the execution end's motion. Therefore, the trajectory data is subjected to discrete cosine transform, filtering, and compression. Data compression is achieved by retaining low-frequency components, while high-frequency noise and outliers are suppressed to form trajectory codes.

[0061] S3: Construct a prediction input sequence based on text encoding, image encoding, and trajectory encoding, and input it into the trajectory prediction model to obtain the first prediction data for the next time period. The control model includes a trajectory prediction model and a strategy generation model. The first prediction data includes the predicted observation image, the predicted text, and a predicted trajectory sequence of a preset length.

[0062] Specifically, the three codes are concatenated and converted into vectors of the same dimension, which facilitates the processing and recognition by the trajectory prediction model.

[0063] S4: Based on the first predicted data and the first actual data, analyze the total cross-entropy loss value of the trajectory prediction model until the total cross-entropy loss value meets the requirements, then stop training the trajectory prediction model.

[0064] Specifically, the trajectory prediction model is first trained, and training can be stopped when the total cross-entropy loss value is less than the preset loss value or when the training times reach the preset number.

[0065] S5: Extract features from language instructions, observed images, and segmented images to obtain multimodal features. The segmented image is a mask image of the target object based on the observed image.

[0066] Specifically, such as Figure 3 As shown, in the second stage, feature extraction is performed on the input information. The second stage introduces new input data: a segmentation image. The segmentation image is a binary image of the same size as the observed image. It is a mask image created by masking the target object to be manipulated on the observed image, highlighting the target object so that the policy generation model can better identify the target object and its location information.

[0067] S6: Input the actual trajectory sequence into the resampler to obtain the actual trajectory features.

[0068] Specifically, during the training phase, to prevent errors from propagating from the trajectory prediction model to the policy generation model and accumulating and amplifying in both models, and to conserve computational resources for the first-stage inference, the predicted trajectory sequence obtained in the first stage is not used as input to the policy generation model. Instead, the actual trajectory sequence from the first stage is directly used as the end effector trajectory supervision signal. This approach avoids the problem of the trajectory prediction model's error gradually amplifying during training, ensuring the accuracy of the supervision signal.

[0069] S7: Construct policy input features based on multimodal features, actual trajectory features, and task status, and input them into the policy generation model to obtain the second prediction data for the next time period. The second prediction data includes the predicted observation image, the predicted task status, and the predicted action parameters of the execution end. The predicted action parameters of the execution end include the predicted values ​​of the robotic arm action and the predicted values ​​of the gripper action.

[0070] S8: Analyze the total loss value based on the second predicted data and the second actual data until the total loss value meets the requirements, then stop training the policy generation model.

[0071] Specifically, the predicted value is compared with the true value to obtain the total loss value. The second true data corresponds to the second predicted data, which includes the observed image at the start of the next time period, the task state, and the action parameters at the end of the execution. When the total loss value is not less than a preset threshold, cyclic training continues. When the total loss value is less than the preset threshold, training stops, and the finally trained policy generation model is put into actual use.

[0072] In this optional embodiment, language commands and observed images are encoded, and then the trajectory data is subjected to discrete cosine transform, filtering, and compression. The obtained text encoding, image encoding, and trajectory encoding are used to construct a prediction input sequence, which is then input into the trajectory prediction model to obtain the first prediction data for the next time period. Based on the first prediction data and the first actual data, the total cross-entropy loss value of the trajectory prediction model is analyzed until the total cross-entropy loss value meets the requirements, at which point training stops and a trained trajectory prediction model is obtained. Then, features are extracted from the language commands, observed images, and segmented images, and the actual trajectory sequence is input into the resampler to obtain actual trajectory features. Multimodal features, actual trajectory features, and task state are used to construct policy input features, which are then input into the policy generation model to obtain the second prediction data for the next time period. Based on the second prediction data and the second actual data, the total loss value is analyzed until the total loss value meets the requirements, at which point training stops and a trained policy generation model is obtained. The two models within the control model are trained separately in stages. When training the policy generation model, actual trajectory data is used instead of the predicted trajectory sequence output by the trajectory prediction model. Training the two models separately in stages avoids the accumulation of errors during training and reduces the training load and data volume of the trajectory prediction model, thus enabling the control model to be obtained quickly. In actual use, the trajectory prediction model in the control model outputs a predicted trajectory sequence of a preset length. Based on this predicted trajectory sequence, the policy generation model outputs the predicted action parameters of the execution end, thereby controlling the motion of the execution end. By designing an independent trajectory prediction model, visual perception and understanding are effectively separated from the generation of action strategies at the execution end, i.e., perception and execution are decoupled. This trajectory prediction model is specifically responsible for parsing language commands and visual observation images, predicting the particle trajectory sequence and the future high-level motion intention of the execution end in the gripper state. Since each output is only a predicted trajectory sequence of a preset length, even if environmental interference or prediction errors occur during control operation, real-time observation images are obtained before the next prediction. The real-time observation images have a certain ability to correct errors and environmental interference during the prediction process, giving the trajectory prediction model self-correcting capabilities. Moreover, the predicted motion trajectory sequence is not directly used to control the execution end, but is used by the strategy generation model to generate the control parameters of the execution end. By constructing a hierarchical, robust, and adaptive control model, this decoupled architecture aims to significantly reduce the propagation and accumulation of perception errors in long action execution links, enabling the system to have stronger stability when facing complex, multi-step, long-term manipulation tasks, and to better generalize to unseen new scenes, new objects, and novel command combinations.

[0073] The following is a detailed description of each step.

[0074] In an optional embodiment of this application, S1: Encoding the language instructions and the observed image to obtain text encoding and image encoding includes:

[0075] S11: Divide the observed image into N image blocks. Each image block is mapped to a vector in the codebook through vector quantization, resulting in a discrete image token for the image block.

[0076] Specifically, the input observed image I is segmented into multiple image patches. Each image block It is a d-dimensional feature vector representing a portion of the observed image. The goal of this process is to transform high-dimensional image data into a set of discrete visual tokens.

[0077] Each image block It will be mapped to the codebook via vector quantization (VQ). The closest vector That is, the discrete image token of the image patch is:

[0078] ;

[0079] in, This represents the discrete image token of the nth image patch. codebook The k-th discrete vector in This represents the nth image patch. In this way, consecutive image patches... It is mapped to discrete visual tokens, thereby realizing the discretization of visual information; Represents norm operations.

[0080] To ensure that the feature vectors of each image patch can be effectively mapped to the codebook while maintaining the quality of the reconstructed image, a total loss function consisting of reconstruction loss and quantization loss is used:

[0081] Reconstruction Loss: Measures the difference between the original image patch I and the reconstructed image patch I. The differences between them ensure that the image information can still be restored after discretization.

[0082] ;

[0083] in, This indicates the losses incurred during reconstruction.

[0084] Quantization loss: encourages the encoder output vector to be close to its corresponding codebook vector, ensuring a stable mapping relationship.

[0085] ;

[0086] Where sg() means to stop gradient propagation to avoid interference from quantization operations on encoder backpropagation; This indicates the quantified loss.

[0087] S12: Perform lexicalization on the language instructions to obtain a text token sequence.

[0088] Specifically, a tokenizer based on a pre-trained model (such as QwenTokenizer) can be used to perform lexicalization on the input multilingual text, transforming the original text into a discrete sequence of tokens. This tokenizer is adaptable to multilingual corpora and has good language versatility and processing capabilities.

[0089] In an optional embodiment of this application, S2: performing discrete cosine transform, filtering, and compression on the trajectory data to obtain trajectory encoding includes:

[0090] S21: Normalize the continuous trajectory data, perform Discrete Cosine Transform (DCT) on each dimension of the normalized trajectory data to obtain DCT coefficients and DCT coefficient sequences. A trajectory value in the trajectory data includes two-dimensional coordinates in the image coordinate system, depth values ​​in the depth map, and end-grip state values.

[0091] Specifically, the input continuous trajectory data is normalized to ensure that the 1st quantile (Q1) and 99th quantile (Q99) of each trajectory dimension are mapped to the range []. [1, 1], thereby eliminating the influence of different action scales, the standardized formula is:

[0092] ;

[0093] Among them, a i This represents the i-th dimension of the trajectory point, where each trajectory point includes four dimensions; This represents the normalized data of the i-th dimension of the trajectory point.

[0094] The normalized trajectory data will then enter the Discrete Cosine Transform (DCT) stage for trajectory encoding. For each trajectory dimension, it is treated as a one-dimensional signal in the time dimension and subjected to DCT processing to extract its frequency components.

[0095] Assuming the time length of the trajectory is H, meaning each trajectory contains H time steps, and each dimension of the trajectory is represented as a scalar sequence. The formula for calculating the k-th DCT coefficient is:

[0096] ;

[0097] in, This represents the value of one dimension of the trajectory at time step n. The DCT coefficients represent the same dimension of the trajectory at the k-th frequency component.

[0098] S22: Select a preset number of DCT coefficients from the beginning of the DCT coefficient sequence and retain them.

[0099] Specifically, by performing DCT on each dimension separately, the trajectory sequence has H time steps, and the trajectory points at each time step have 4 dimensions. This refers to any one of the dimensions, for example, for The dimension column, assuming its original data is After DCT formula transformation, it is converted into discrete frequency domain coefficients. Because trajectories typically exhibit high temporal correlation, with most information concentrated in the low-frequency components, only the first N1 low-frequency coefficients are usually retained during encoding. This method effectively compresses and removes high-frequency noise. It not only reduces the length of the trajectory representation but also provides a sparsity basis for subsequent BPE encoding, contributing to the generation of stable and efficient discrete token sequences. The DCT transformation yields a sparse matrix where most high-frequency components are zero. To further compress the trajectory representation, only the low-frequency coefficients with significant impact are retained, reducing dimensionality and improving model training efficiency.

[0100] S23: The retained DCT coefficients are quantized according to the preset quantization precision and mapped to discrete symbols. Then, BPE is introduced for encoding to generate a trajectory token sequence.

[0101] Specifically, the BPE compression strategy is adopted to further reduce the storage space of the sparse matrix. The BPE vocabulary size is 1024. The retained DCT coefficients are quantized according to a preset quantization precision and mapped to a discrete symbol sequence. For example, if the DCT coefficient is 2.36 and the preset quantization precision is 0.1, then 2.36 is quantized to 2.4, and then mapped to the corresponding discrete symbol (e.g., D1). Then, BPE is introduced for encoding to discretize the continuous frequency data into a trajectory token sequence.

[0102] The trajectory data processed as described above is transformed into a discrete token sequence. Each This represents a discrete trajectory marker (i.e., a point). In this way, each execution step in the trajectory is encoded as a discrete marker, which is then adapted to deep learning models for modeling and prediction.

[0103] In an optional embodiment of this application, S3: A prediction input sequence is constructed based on text encoding, image encoding, and trajectory encoding, and input into the trajectory prediction model to obtain the first prediction data for the next time period. The first prediction data includes the predicted observation image, the predicted text, and a predicted trajectory sequence of a preset length.

[0104] Specifically, the trajectory prediction model primarily employs an autoregressive language model, extending the embedding layer structure to handle text tokens and further supporting trajectory and visual tokens. To achieve unified modality modeling, the model first organizes the three types of tokens into a unified input sequence. Each modality is separated by explicit start and end markers, ensuring the trajectory prediction model accurately identifies modality boundaries and models their internal structure. Each modality uses a separate set of prefix and end markers, resulting in the following predicted input sequence structure:

[0105]

[0106] in, and These represent the start and end markers of a text modality, respectively. and These represent the start and end markers of the image modality, respectively; and Indicates the start and end markers of the trajectory mode; A sequence of text tokens; A sequence of image tokens; For trajectory token sequence, This indicates a concatenation operation, which involves combining multiple types of token sequences together.

[0107] Thanks to the guiding role of modality prefixes and end tokens, the model can learn the contextual interactions between different modalities while maintaining structural consistency within each modality. For each token in the above input, it is transformed into a vector representation of uniform dimension through independent embedding mapping layers. These embedding layers share a set of positional encoding mechanisms, enabling the model to capture the sequential structure in the token sequence.

[0108] ;

[0109] in, This indicates the token type, which can be text, image, or track. This represents the input vector of the embedding layer. Indicates that the token type is hour, The i-th token in the input sequence of type , This indicates the position of the token in the predicted input sequence.

[0110] In an optional embodiment of this application, S4: Based on the first predicted data and the first actual data, analyze the total cross-entropy loss value of the trajectory prediction model until the total cross-entropy loss value meets the requirements, and then stop training the trajectory prediction model.

[0111] Specifically, the entire model is trained in an autoregressive manner, with the training objective of maximizing the predicted probability of the next token. The loss function is the standard cross-entropy loss. To balance the training contributions of different modalities, a modality-weighted strategy is introduced. The total cross-entropy loss value is:

[0112]

[0113] in, CE() represents the total cross-entropy loss value, and CE() represents the cross-entropy loss function. , and These represent the cross-entropy loss values ​​for the text, the observed image, and the trajectory, respectively. The weights represent the cross-entropy loss values ​​of the observed images. The weights representing the cross-entropy loss values ​​of the trajectory can, for example, be set to... , .

[0114] In an optional embodiment of this application, S5: Feature extraction is performed on the language instructions, the observed image, and the segmented image to obtain multimodal features, including:

[0115] S51: A CLIP-based text encoder is used to extract features from language instructions to obtain text features. .

[0116] Specifically, in terms of language information processing, the text encoding branch of a CLIP-based encoder is used to extract features from the input task language instructions. This encoder outputs a fixed-length language feature vector, which serves as the language representation for subsequent multimodal alignment. .

[0117] S52: Input the observed image and the segmented image (i.e., the image of the object to be operated on, such as the image of a ball that needs to be moved) into the CLIP-based image encoder to obtain the feature set of the observed image. and segmentation image feature set .

[0118] Specifically, in the visual information processing section, the initial observation frame and segmentation map The input is fed into the visual encoding branch of the CLIP-based encoder. The CLIP encoder is pre-trained on large-scale image-text alignment data, effectively extracting information about target objects and scene layout in images. Through the CLIP encoder, a set of patch-level visual features is obtained:

[0119] , ,

[0120] in, This represents the total number of image blocks into which the observed image is divided; This indicates the total number of image blocks into which the segmented image is divided; This represents the visual feature vector of the i-th image patch in the observed image; This represents the visual feature vector of the i-th image patch in the segmented image; Represents the set of features of the observed image; This represents the set of features in a segmented image.

[0121] S53: Perform global feature compression on the observed image and the segmented image to obtain the global feature vector of the observed image. and segmentation of global feature vectors of images .

[0122] Specifically, after compressing the global features of the observed image and the segmented image, the global feature vectors of the corresponding images are obtained respectively:

[0123] , .

[0124] S54: Concatenate the text features, the observation image feature set, the segmentation image feature set, the global feature vector of the observation image, and the global feature vector of the segmentation image to obtain multimodal features. .

[0125] Specifically, then, at the channel dimension, the language features are concatenated with two sets of visual global and local features to form a unified multimodal feature:

[0126] ,

[0127] Concat() represents a concatenation operation along the feature dimension.

[0128] In an optional embodiment of this application, S6: The actual trajectory sequence is input to the resampler to obtain the actual trajectory features.

[0129] Specifically, in the multimodal fusion process, if the complete actual trajectory sequence is directly used... Injecting a policy network incurs high computational costs, especially when H=10, and the computational complexity of the model becomes O(n log n). This means that the computational complexity increases by the square of H. To address this, a trajectory resampling module was designed, which uses a set of trainable query vectors. (in (where is the fixed number of particles after resampling), and the actual trajectory features are obtained by querying the fully labeled trajectory through interactive attention:

[0130] ,

[0131] in, This represents the interactive attention mechanism in the resampler, where q represents the query vector in the resampler, and represents the number of trajectory particles (trajectory points) that need to be extracted from the most suitable complete trajectory. Indicates the first The position vector at the end of the execution of a time step. This represents the actual trajectory sequence of length H. , and These represent the two-dimensional coordinates of the execution endpoint in the image coordinate system, with the top-left corner of the image as the origin. Represents the depth value of the depth map; This indicates the gripper state; -1 indicates closed, and 1 indicates open.

[0132] In an optional embodiment of this application, S7: A policy input feature is constructed based on multimodal features, actual trajectory features, and task state, and input into the policy generation model to obtain the second prediction data for the next time period. The second prediction data includes the predicted observation image, the predicted task state, and the predicted action parameters of the execution end.

[0133] Specifically, existing models lack the ability to explicitly model and judge the execution state of a task. They typically assume that instructions and the environment are always matched and executable. However, when encountering situations where instructions are ambiguous, conflict with the environment, or the current state cannot meet the requirements of the instructions, the model may produce invalid or even dangerous actions, lacking robustness and safety.

[0134] To address the mismatch between language instructions and tasks, task state is introduced when constructing the policy generation model. This enhances the effectiveness of language conditions in constraining robot behavior generation. Task state is an additional auxiliary dimension added to the output space, namely, task state features. This state is used to characterize the current execution status of a task. The task state is a discrete variable with three possible values: execution state S1; completion state S2; and abnormal state S3. Execution state S1 indicates that the task is being executed; completion state S2 indicates that the task has successfully ended; and abnormal state S3 indicates that, under the current environmental perception, the language instructions are not executable, and is used to identify scenarios where there is a conflict or inconsistency between the environment and the instructions.

[0135] During training, the following methods are used to enhance the model's generalization ability and semantic consistency judgment ability: randomly select some samples and replace their language instructions with invalid instructions that do not match the scene to form abnormal samples; for the above samples, only supervise their task state output to S3, and do not apply loss to other action dimensions (such as pose, gripper, etc.); this mechanism guides the model to learn the matching relationship between language instructions and environmental perception, thereby improving its task discrimination and fault tolerance in complex scenes.

[0136] In this embodiment, explicit task state modeling introduces discrete task state variables, giving the model the ability to judge the feasibility of language instructions and the progress of the task. This not only helps the model stop naturally after the task is completed, but more importantly, it can identify semantic conflicts between language instructions and environmental awareness, thereby avoiding the execution of invalid or even dangerous actions, significantly enhancing the system's security, adaptability, and decision-making rationality in open and dynamic environments.

[0137] Furthermore, existing strategy models typically employ fixed multimodal fusion methods, such as simple feature concatenation or attention weighting, lacking a mechanism to dynamically adjust the importance of different modalities based on the task execution stage. When high-precision operations are required (such as grasping and placing), over-reliance on coarse trajectory prediction may lead to failure; while for long-distance movement, over-reliance on visual details is inefficient. Trajectory guidance and visual guidance can effectively help robots quickly approach target objects, and when approaching or contacting the task object, visual guidance can provide higher precision for grasping or releasing operations. Therefore, a dynamic model weight switching mechanism is designed in the model to organically combine trajectory guidance branches and image guidance branches under different conditions.

[0138] Specifically, for each sample's predicted trajectory sequence, the gripper state is detected in the last three time steps. Has it changed: If it changes from fully open in the three time steps To fully close If the state changes to the opposite direction, it is determined that the robotic arm is about to perform a critical operation (grasping or releasing). At this point, the weights of the trajectory-guided branches are adjusted. Set it to 0, and simultaneously adjust the weight of the visual guidance branch to... .

[0139] Finally, the complete policy input features are:

[0140] ,

[0141] in, Represents the weight of the trajectory-guided branch; regarding the weight of the trajectory-guided branch Dynamic switching mechanism:

[0142] .

[0143] The weight of the trajectory-guided branch is This ensures that the model smoothly transitions from trajectory-driven to fine-grained visual control as it approaches the operating range.

[0144] In this embodiment, the dynamic weight switching mechanism adaptively adjusts the weights of trajectory guidance and visual guidance based on the task execution stage, particularly by detecting changes in the gripper state to predict key operations such as grasping / releasing. During long-distance movement, efficient trajectory guidance is emphasized, while in the contact / interaction stage requiring high-precision operations, a smooth transition to detailed visual guidance is achieved, thereby significantly improving the overall success rate and operational accuracy of task execution.

[0145] In an optional embodiment of this application, S8: Analyze the total loss value based on the second predicted data and the second actual data until the total loss value meets the requirements, and then stop training the policy generation model.

[0146] Specifically, the total loss value is

[0147] ,

[0148] in, Indicates loss of action, Indicates the loss of the observed image. This indicates a loss in task status.

[0149] The motion loss includes the robot's robotic arm motion loss, gripper motion loss, and KL divergence loss.

[0150] KL divergence loss:

[0151] ,

[0152] in, This represents the mean of the predicted values ​​for the robotic arm's movements. represents the standard deviation of the predicted values ​​of the robotic arm's movements; ln() represents the natural logarithm function.

[0153] Robotic arm motion loss:

[0154]

[0155] in, This is the predicted value for the robotic arm's movements. The actual value of the robotic arm's movements. is the number of samples; SmoothL1Loss() represents the smoothed absolute loss.

[0156] Gripper motion loss:

[0157]

[0158] in, The predicted value of the gripper's action after Sigmoid activation. This represents the actual value of the gripper's action.

[0159] ,

[0160] in, The weights representing the gripper motion loss, for example .

[0161] like Figure 4 As shown in the figure, an embodiment of this application provides a robot long-term operation control method, including:

[0162] S10: Encode the obtained language instructions, observation images, and initial position information of the execution end to obtain the predicted input sequence.

[0163] Specifically, following the methods described in S1-S3 above, the language instructions, the observed images, and the initial position information of the execution end are encoded to obtain the corresponding token sequences. Multiple sequences are then combined in order and segmented by adding markers to obtain the predicted input sequence.

[0164] S20: Input the predicted input sequence into the trajectory prediction model within the control model obtained by the above-mentioned robot long-term operation control model training method to obtain a predicted trajectory sequence of a preset length.

[0165] Specifically, the trajectory prediction model not only outputs a predicted trajectory sequence of a preset length, but also outputs the predicted observation image. However, the predicted observation image output at this stage is a process quantity, which can be used during training to calculate the model's loss value. In practical applications, this prediction quantity is not used.

[0166] S30: Extract features from language instructions, observed images, segmented images, and predicted trajectory sequences of a preset length to obtain policy input features.

[0167] S40: Determine whether the gripper state remains consistent within the last specified number of steps in the predicted trajectory sequence of the preset length; if it remains consistent, maintain the weight of the trajectory guidance branch in the policy input feature unchanged; otherwise, reset the weight of the trajectory guidance branch in the policy input feature to zero.

[0168] Specifically, following S5 above, multimodal features are obtained. Then, similar to S6 above, a predicted trajectory sequence of a preset length is input to the resampler to obtain predicted trajectory features. Based on the multimodal features, predicted trajectory features, and the current task state, policy input features are constructed. Before inputting these features into the policy generation model, the gripper state within the last specified number of steps in the predicted trajectory sequence is monitored and judged to identify whether the robot's end effector is close to the target object, thereby changing the movement guidance strategy. Additionally, regarding the weights of the trajectory guidance branches... The dynamic switching mechanism switches in the following manner.

[0169] .

[0170] S50: Input the policy input features into the policy generation model within the control model obtained by the above-mentioned robot long-term operation control model training method to obtain the predicted action parameters and predicted task state of the execution end, and use the predicted action parameters to control the robot's execution end work according to the predicted task state.

[0171] Specifically, the strategy generation model outputs predicted observation images, predicted task states, and predicted motion parameters. The predicted motion parameters include movement parameters and gripper states, namely the translational motion of the robotic arm (translation in three dimensions) and the rotational motion of the end effector (rotation in three dimensions). Combined with the gripper states, the predicted motion parameters constitute a 7-dimensional dataset. When the predicted task state is execution state S1, the predicted motion parameters are used to control the robot's end effector; when the predicted task state is completion state S2, the robot's end effector stops working; and when the predicted task state is abnormal state S3, the robot's end effector is interrupted, and a warning is issued.

[0172] In summary, compared with existing technologies, it has the following beneficial effects:

[0173] 1. This application proposes a method for hierarchical processing of trajectory prediction and policy generation. By first generating a future trajectory sequence and using it as input to guide the subsequent policy generation process, efficient separation and collaborative optimization of trajectory and policy are achieved. Utilizing a hierarchical planning scheme for trajectory guidance and policy generation, trajectory generation and policy execution are decoupled, significantly improving the robot's execution accuracy and response speed in long-term manipulation tasks. This hierarchical planning scheme enables the robot to predict trajectories and adjust policies in real time, effectively coping with complex tasks in dynamic environments.

[0174] 2. This application designs a trajectory resampling mechanism based on query vectors. By learning the query vectors, important low-frequency features are extracted from the complete trajectory, significantly reducing the computational overhead caused by long trajectory inputs. Simultaneously, a dynamic weight switching mechanism is proposed, adaptively adjusting the weights of trajectory guidance and visual guidance according to changes in the gripper state. This ensures that the robot can smoothly transition to higher-precision visual control when performing fine tasks such as grasping, thus improving operational accuracy.

[0175] 3. This application introduces a task state variable as an auxiliary dimension to represent the current execution status of the task. Through the task state judgment mechanism, the model can better handle semantic and environmental conflicts during task execution, improving task discrimination and fault tolerance in complex environments.

[0176] like Figure 5 As shown in the figure, an embodiment of this application provides a robot long-term operation control system, comprising:

[0177] The encoding module 10 is used to encode the obtained language instructions, observation images and initial position information of the execution end respectively to obtain the predicted input sequence.

[0178] The trajectory prediction module 20 is used to input the predicted input sequence into the trajectory prediction model within the control model obtained by the above-mentioned robot long-term operation control model training method, and obtain a predicted trajectory sequence of preset length.

[0179] The feature extraction module 30 is used to extract features from language instructions, observed images, segmented images, and predicted trajectory sequences of a preset length to obtain policy input features.

[0180] The weight switching module 40 is used to determine whether the gripper state remains consistent within the last specified number of steps in the predicted trajectory sequence of the preset length. If it remains consistent, the weight of the trajectory guidance branch in the policy input feature is kept unchanged; otherwise, the weight of the trajectory guidance branch in the policy input feature is reset to zero.

[0181] The strategy generation module 50 is used to input the strategy input features into the strategy generation model within the control model obtained by the above-mentioned robot long-term operation control model training method, to obtain the predicted action parameters of the execution end, and to control the robot's execution end to work according to the predicted task state.

[0182] It also includes a training module 60, which is used to execute the above-mentioned robot long-term operation control model training method to obtain the control model.

[0183] In this embodiment, the beneficial effects of the robot long-term operation control system are similar to those of the robot long-term operation control method described above, and will not be repeated here.

[0184] An electronic device provided in this application includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement the robot long-term operation control method or the robot long-term operation control model training method described above when the computer program is executed.

[0185] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the robot long-term operation control method or the robot long-term operation control model training method described above.

[0186] In this embodiment, the beneficial effects of the electronic device and the computer-readable storage medium are similar to those of the robot long-term operation control method described above, and will not be repeated here.

[0187] The present invention describes electronic devices that can serve as servers or clients of this application, which are examples of hardware devices that can be applied to various aspects of this application. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the application described and / or claimed herein.

[0188] Electronic devices include a computing unit that can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM can also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0189] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0190] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0191] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for training a long-term operation control model for a robot, characterized in that, include: The language instructions and observed images are encoded to obtain text encoding and image encoding; The trajectory data is subjected to discrete cosine transform, filtered, and compressed to obtain the trajectory code; The prediction input sequence is constructed based on text encoding, image encoding and trajectory encoding, and then input into the trajectory prediction model to obtain the first prediction data for the next time period. The control model includes a trajectory prediction model and a strategy generation model. The first prediction data includes the predicted observation image, the predicted text and the predicted trajectory sequence of a preset length. Based on the first predicted data and the first actual data, analyze the total cross-entropy loss value of the trajectory prediction model until the total cross-entropy loss value meets the requirements, and then stop training the trajectory prediction model. Multimodal features are obtained by extracting features from language instructions, observed images, and segmented images. The segmented image is a mask image of the target object based on the observed image. The actual trajectory sequence is input into the resampler to obtain the actual trajectory features; Based on multimodal features, actual trajectory features, and task status, policy input features are constructed and input into the policy generation model to obtain the second prediction data for the next time period. The second prediction data includes the predicted observation image, the predicted task status, and the predicted action parameters of the execution end. The predicted action parameters of the execution end include the predicted values ​​of the robotic arm action and the predicted values ​​of the gripper action. Based on the second predicted data and the second actual data, analyze the total loss value until the total loss value meets the requirements, then stop training the policy generation model.

2. The robot long-term operation control model training method as described in claim 1, characterized in that, The process of encoding language instructions and observed images to obtain text encoding and image encoding includes: The observed image is divided into N image blocks. Each image block is mapped to a vector in the codebook through vector quantization, resulting in a discrete image token for that image block: ； in, This represents the discrete image token of the nth image patch. codebook The k-th discrete vector in This represents the nth image patch; Represents norm operations; The language instructions are lexicalized to obtain a text token sequence.

3. The robot long-term operation control model training method as described in claim 1, characterized in that, The process of performing discrete cosine transform, filtering, and compression on the trajectory data to obtain the trajectory code includes: The continuous trajectory data is normalized, and the discrete cosine transform is performed on each dimension of the normalized trajectory data to obtain DCT coefficients and DCT coefficient sequences. A trajectory value in the trajectory data includes two-dimensional coordinates in the image coordinate system, depth values ​​in the depth map, and end-gripper state values. Select a preset number of DCT coefficients from the beginning of the DCT coefficient sequence and retain them; The retained DCT coefficients are quantized according to a preset quantization precision and mapped to discrete symbols. Then, BPE is introduced for encoding to generate a trajectory token sequence.

4. The robot long-term operation control model training method as described in claim 1, characterized in that, The total cross-entropy loss value: ； in, CE() represents the total cross-entropy loss value, and CE() represents the cross-entropy loss function. , and These represent the cross-entropy loss values ​​for the text, the observed image, and the trajectory, respectively. The weights represent the cross-entropy loss values ​​of the observed images. The weights represent the cross-entropy loss values ​​of the trajectories.

5. The robot long-term operation control model training method as described in claim 1, characterized in that, The feature extraction from language instructions, observed images, and segmented images to obtain multimodal features includes: A CLIP-based text encoder is used to extract features from language instructions to obtain text features. ; The observed image and the segmented image are input into a CLIP-based image encoder to obtain the feature set of the observed image. and segmentation image feature set ; ， ； in, This represents the total number of image blocks into which the observed image is divided; This indicates the total number of image blocks into which the segmented image is divided; This represents the visual feature vector of the i-th image patch in the observed image; This represents the visual feature vector of the i-th image patch in the segmented image; Represents the set of features of the observed image; Represents the set of features in a segmented image; Global feature compression is performed on the observed image and the segmented image to obtain the global feature vector of the observed image. and segmentation of global feature vectors of images ; The text features, the set of features of the observed image, the set of features of the segmented image, the global feature vector of the observed image, and the global feature vector of the segmented image are concatenated to obtain the multimodal features. .

6. The robot long-term operation control model training method as described in claim 5, characterized in that, The actual trajectory features are: ； in, This represents the interactive attention mechanism in the resampler, where q represents the query vector in the resampler. Indicates the first The position vector at the end of the execution of a time step. This represents the actual trajectory sequence of length H. , and These represent the two-dimensional coordinates of the execution endpoint in the image coordinate system, with the top-left corner of the image as the origin. Represents the depth value of the depth map; This indicates the gripper state; -1 indicates closed, and 1 indicates open.

7. The robot long-term operation control model training method as described in claim 6, characterized in that, The policy input feature is: ； in, This indicates the weight of the trajectory-guided branch. Represents multimodal features, Represents the task state features; Concat() represents the concatenation operation along the feature dimension. Weights of trajectory-guided branches The dynamic switching mechanism is 。 8. The robot long-term operation control model training method as described in claim 1, characterized in that, The total loss value is ； in, Indicates loss of action, This represents the loss of the observed image. Indicates the loss of task status; The motion loss includes the robot's robotic arm motion loss, gripper motion loss, and KL divergence loss; KL divergence loss: ； in, This represents the mean of the predicted values ​​for the robotic arm's movements. The standard deviation of the predicted values ​​of the robotic arm's movements is represented by ln(); ln() represents the natural logarithm function. Robotic arm motion loss: in, This is the predicted value for the robotic arm's movements. N represents the true value of the robotic arm's movements, and N is the number of samples; SmoothL1Loss() represents the smoothed absolute value loss; Gripper motion loss: in, The predicted value of the gripper's action after Sigmoid activation. This represents the actual value of the gripper's action. ； in, This represents the weight of the gripper motion loss.

9. A method for controlling long-term operation of a robot, characterized in that, include: The obtained language instructions, observation images, and initial position information of the execution end are encoded to obtain the predicted input sequence; The predicted input sequence is input into the trajectory prediction model within the control model obtained by the robot long-term operation control model training method according to any one of claims 1-8, to obtain a predicted trajectory sequence of a preset length; Features are extracted from language instructions, observed images, segmented images, and predicted trajectory sequences of a preset length to obtain policy input features; Determine whether the gripper state remains consistent within the last specified number of steps in the predicted trajectory sequence of the preset length; if it remains consistent, maintain the weight of the trajectory guidance branch in the policy input features unchanged; otherwise, reset the weight of the trajectory guidance branch in the policy input features to zero. The strategy input features are input into the strategy generation model within the control model obtained by the robot long-term operation control model training method according to any one of claims 1-8, to obtain the predicted action parameters of the execution end, and the robot's execution end is controlled by the predicted action parameters according to the predicted task state.

10. A long-term operation control system for a robot, characterized in that, include: The encoding module is used to encode the obtained language instructions, observation images, and initial position information of the execution end to obtain the predicted input sequence. The trajectory prediction module is used to input the predicted input sequence into the trajectory prediction model within the control model obtained by the robot long-term operation control model training method according to any one of claims 1-8, and obtain a predicted trajectory sequence of a preset length. The feature extraction module is used to extract features from language instructions, observed images, segmented images, and predicted trajectory sequences of a preset length to obtain policy input features; The weight switching module is used to determine whether the gripper state remains consistent within the last specified number of steps in the predicted trajectory sequence of a preset length. If it remains consistent, the weight of the trajectory guidance branch in the policy input feature is kept unchanged; otherwise, the weight of the trajectory guidance branch in the policy input feature is reset to zero. The strategy generation module is used to input the strategy input features into the strategy generation model within the control model obtained by the robot long-term operation control model training method according to any one of claims 1-8, to obtain the predicted action parameters of the execution end, and to control the robot's execution end to work according to the predicted task state using the predicted action parameters.

Images