Robot control method, device and system and computer readable storage medium
By processing visual-language information through cascaded attention modules, the problem of inaccurate prediction of action sequences in complex scenarios by visual-language-action models is solved, and more accurate and stable robot motion control is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- DAIMON (SHENZHEN) ROBOTICS TECHNOLOGY CO LTD
- Filing Date
- 2026-03-27
- Publication Date
- 2026-05-15
AI Technical Summary
Existing vision-language-action models are inaccurate in predicting action sequences in complex scenes, which affects the performance of robot tasks.
Multiple cascaded attention modules are used to process information from different modalities. Through multiple attention layers, visual features, query features, and action latent variables are fused across modalities and self-fused to ensure the effective use of information from each modality and avoid over-reliance on a single modality.
It improves the accuracy and stability of action sequence prediction and enhances the robot's ability to model visual details and task objectives.
Smart Images
Figure CN122033972A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of robots, and in particular to a robot control method, device, system and computer readable storage medium. BACKGROUND
[0002] A Vision-Language-Action (VLA) model is a common model in the field of robot control. Through the VLA model, natural language instructions and visual observation images can be mapped into robot control signals, so that the robot can handle complex tasks.
[0003] The VLA model currently used predicts the action sequence of the robot by cross-modal information fusion of natural language instructions and visual observation images. This way uses highly aggregated cross-modal fusion features for robot action sequence generation, which is prone to inaccurate action sequence generation in complex scenarios, thereby affecting the effect of the robot executing tasks. SUMMARY
[0004] The present application provides a robot control method, device, system and computer readable storage medium, which solves the problem of inaccurate action sequence prediction in the prior art.
[0005] In a first aspect, an embodiment of the present application provides a robot control method, the method comprising:
[0006] processing the obtained target image and robot control instruction to obtain visual features and query features; inputting the visual features, the query features and an initial action latent variable into a plurality of cascaded attention layers to obtain a target action latent variable; each of the plurality of cascaded attention layers comprises a first attention module, a second attention module, a third attention module and a feedforward network module; wherein the first attention module is configured to perform cross-modal fusion on a current action latent variable and the query features to obtain first joint features; the second attention module is configured to perform self-fusion on the current action latent variable to obtain a fused action latent variable; the third attention module is configured to perform cross-modal fusion on the current action latent variable and the visual features to obtain second joint features; the feedforward network module is configured to obtain an updated action latent variable according to the first joint features, the fused action latent variable and the second joint features; the current action latent variable is an updated action latent variable obtained after at least one update of the initial action latent variable; the current action latent variable corresponding to the first attention layer of the plurality of cascaded attention layers is the initial action latent variable; the updated action latent variable obtained by the last attention layer of the plurality of cascaded attention layers is the target action latent variable. The latent variables of the target action are decoded to obtain the action sequence, and the robot is controlled to perform the corresponding action based on the action sequence.
[0007] Optionally, the cross-modal fusion of the current action latent variable and the visual features includes: A third query vector is obtained based on the current action latent variables; a third key vector and a third value vector are obtained based on the visual features; The third weight matrix is obtained based on the third query vector and the third key vector; The top M elements with the highest values in each row of the third weight matrix are determined, and the remaining elements are set to preset masking values to obtain the fourth weight matrix; where M is a natural number greater than 0. The second joint feature is obtained by weighting and fusing the third value vector based on the fourth weight matrix.
[0008] Optionally, the step of weighting and fusing the third value vector based on the fourth weight matrix to obtain the second joint feature includes: The third value vector is weighted and fused based on the fourth weight matrix to obtain visual attention features; The current action latent variable is subjected to linear transformation and nonlinear activation to obtain the gating coefficients; The corresponding gating coefficient is multiplied element-wise with the visual attention feature to obtain the second joint feature.
[0009] Optionally, the updated action latent variable obtained based on the first joint feature, the fused action latent variable, and the second joint feature includes: The first joint feature, the fused action latent variable, and the second joint feature are concatenated to obtain the concatenated feature; The spliced features are subjected to linear transformation and nonlinear activation to obtain refined features; The refined features and the current action latent variables are added element by element to obtain the updated action latent variables.
[0010] Optionally, the processing of the acquired target image and robot control commands to obtain visual features and query features includes: The acquired target image is encoded to obtain visual features; The robot control commands are processed to obtain command characteristics; The visual features and the instruction features are fused to obtain the query features.
[0011] Optionally, the target image includes a global image and a local image of the wrist, and the encoding of the acquired target image to obtain visual features includes: Feature extraction is performed on the acquired global image and wrist image respectively to obtain global visual features and local visual features; The visual features are obtained by concatenating the global visual features and the local visual features.
[0012] Secondly, embodiments of this application provide a robot control device, the device comprising: The acquisition module is configured to process the acquired target image and robot control commands to obtain visual features and query features; An attention module is configured to derive a target action latent variable based on the input visual features, the query features, and an initial action latent variable. The attention module comprises multiple cascaded attention layers, each of which includes a first attention module, a second attention module, a third attention module, and a feedforward network module. Specifically, the first attention module is configured to perform cross-modal fusion on the current action latent variable and the query features to obtain a first joint feature; the second attention module is configured to perform self-fusion on the current action latent variable to obtain a fused action latent variable; the third attention module is configured to perform cross-modal fusion on the current action latent variable and the visual features to obtain a second joint feature; the feedforward network module is configured to obtain an updated action latent variable based on the first joint feature, the fused action latent variable, and the second joint feature; the current action latent variable is the updated action latent variable obtained after at least one update of the initial action latent variable; the current action latent variable corresponding to the first attention layer in the multiple cascaded attention layers is the initial action latent variable; and the updated action latent variable obtained by the last attention layer in the multiple cascaded attention layers is the target action latent variable. The decoding module is configured to decode the latent variables of the target action to obtain an action sequence, and control the robot to perform the corresponding action based on the action sequence.
[0013] Thirdly, embodiments of this application provide a robot control device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the robot control method described above.
[0014] Fourthly, embodiments of this application provide a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program, when executed by a processor, implements the steps of the robot control method described above.
[0015] Fifthly, embodiments of this application provide a robot control system, which includes a robot and a robot control device; the robot is equipped with a camera for acquiring target images; the robot control device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the robot control method described above.
[0016] The robot control method provided in this application processes information from different modalities separately through cascaded attention modules, thereby extracting features from each modality more fully and ensuring that information from each modality can be effectively utilized. This prevents the neglect of information from any particular modality in the multimodal information and avoids the model from over-relying on information from any one modality. In this way, the model can utilize multimodal information more evenly, improve its ability to model visual details and task objectives, and enhance the accuracy of action sequence prediction. Attached Figure Description
[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a schematic diagram of a robot control system according to an embodiment of this application; Figure 2 This is a flowchart illustrating the robot control method according to an embodiment of this application; Figure 3 A topology diagram of multiple attention layers; Figure 4 This is a schematic diagram of the structure of a robot control device according to an embodiment of this application. Detailed Implementation
[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.
[0021] To facilitate understanding, the application scenarios involved in this application will be explained and described below. The robot control method provided in this application is applied to robot motion control. The robot can be an end effector, a robotic arm including an end effector, or a unibody intelligent robot; the embodiments of this application do not limit the specific form of the robot. The robot control method provided in this application can be applied to scenarios including but not limited to: specific task execution, such as object grasping. The following embodiments will be described in detail using the object grasping scenario as an example.
[0022] Figure 1 A schematic diagram of a robot control system according to an embodiment of this application is shown. The robot control system includes a robot control device 100 and a robot 200, which are communicatively connected. A camera is mounted on the robot 200 for acquiring target images. In some embodiments, the camera on the robot 200 is configured to acquire a global image including the end effector and the object to be grasped. In other embodiments, the camera on the robot 200 is configured to acquire a local image including the end effector and the object to be grasped. The local image may reflect the current posture of the end effector and the relative positional relationship between the end effector and the object to be grasped. It should be understood that in specific implementations, the robot 200 can be configured to acquire at least one of the global and local images. The robot control device 100 is configured to receive images acquired by the robot 200 for further image processing. The robot control device 100 is also configured to receive language instructions, which are voice or text instructions input by a user in natural language, describing the grasping task the robot needs to complete. For example, in a specific embodiment, the language instruction could be "grab the water glass on the table." The aforementioned images and verbal commands form the basis for generating robot motion sequences. The robot's motion sequences are obtained based on the input images and verbal commands.
[0023] Existing Vision-Language-Action (VLA) models typically use an attention module to handle the interaction and fusion of multimodal information when generating action sequences. This fusion method disperses attention weights, preventing the model from fully exploring local features of objects, subtle positional changes, and key operational regions in the image. As a result, visual detail information is diluted during cross-modal fusion, ultimately leading to problems such as biased action understanding and inaccurate action sequence prediction in complex scenes, affecting the accuracy and stability of task execution.
[0024] The robot control method provided in this application processes information from different modalities separately through cascaded attention modules, thereby extracting features from each modality more fully and ensuring that information from each modality can be effectively utilized. This prevents the neglect of information from any particular modality in the multimodal information and avoids the model from over-relying on information from any one modality. In this way, the model can utilize multimodal information more evenly, improving its ability to model visual details and task objectives, as well as the accuracy of action sequence prediction.
[0025] The embodiments of this application are described below.
[0026] Figure 2 A flowchart illustrating a robot control method provided in an embodiment of this application. (Refer to...) Figure 2 The method includes the following steps: Step S10: Process the acquired target image and robot control instructions to obtain visual features and query features.
[0027] In this step, the target image is an image related to the task performed by the robot, and the robot control command is a language command related to the task performed by the robot. The target image can be captured by a camera installed on the robot body, and the device executing this method acquires the target image captured by the camera. Taking the target object grasping task as an example, the target image is an image including the target object, such as at least one of the global image and local image mentioned above. The robot control command is a language command input by the user, which can be in text or voice form.
[0028] Processing the target image includes: encoding the target image using an image encoder to obtain visual features. The image encoder can be any type of neural network module used to map a high-dimensional image into a low-dimensional visual feature vector. In this embodiment, the image encoder can be a Vision Transformer model or any other convolutional neural network model; this embodiment is not limited to a specific type of image encoder. Processing the control commands includes: encoding the control commands using a language encoder to obtain command features. The language encoder can be any type of text encoding network, such as an LLM text encoding network; this embodiment is not limited to a specific type of language encoder. Processing the target image and robot control commands further includes: fusing the visual features and command features to obtain query features. In some embodiments, the visual features and command features are concatenated to obtain concatenated features, and the concatenated features are linearly mapped to obtain query features. Specifically, the visual features and command features are concatenated along the feature dimension; for example, if the visual features are 768-dimensional and the command features are 512-dimensional, then the concatenated features obtained after concatenation are 1280-dimensional. The concatenated features are linearly mapped through a linear layer to obtain the query features. This linear layer consists of a learnable weight matrix and a bias matrix. In this embodiment, the linear layer can be based on any neural network architecture, and this application is not limited to this. In other embodiments, visual features and instruction features are each linearly projected, and their similarity is aligned to obtain the query features. Specifically, when linearly projecting the visual and instruction features, both features are mapped to the same dimensional space. The dot product of the projected visual and instruction features is calculated to obtain their similarity. The similarity is normalized, and the normalized similarity is weighted and fused with the projected visual features to align the visual and instruction features, thus obtaining the query features. The above method for determining query features is merely an example and does not constitute a limitation on the method for determining query features.
[0029] Step S20: Input the visual features, query features, and preset initial action latent variables into multiple cascaded attention layers to obtain the target action latent variables.
[0030] In this step, multiple attention layers are connected in a cascaded manner. The output of the previous attention layer serves as the input to the next attention layer connected to it. The action latent variables are updated through multiple cascaded attention layers to obtain the target action latent variables. In this step, each attention layer includes multiple attention modules. The number of attention modules in each layer can be set by those skilled in the art based on the number of input features of the multiple attention layers. In this embodiment, each attention layer includes three attention modules. Assuming the number of attention layers is N, the specific topology of N attention layers is as follows... Figure 3 As shown. In Figure 3 In this architecture, each attention layer comprises three attention modules and one feedforward network module. The first attention module is configured to perform cross-modal fusion of the current action latent variable and query features to obtain a first joint feature; the second attention module is configured to perform self-fusion of the current action latent variable to obtain a fused action latent variable; and the third attention module is configured to perform cross-modal fusion of the current action latent variable and visual features to obtain a second joint feature. The feedforward network module is configured to obtain an updated action latent variable based on the first joint feature, the fused action latent variable, and the second joint feature. The current action latent variable is the updated action latent variable obtained after at least one update of the initial action latent variable; the current action latent variable corresponding to the first attention layer in a series of cascaded attention layers is the initial action latent variable; and the updated action latent variable obtained from the last attention layer in a series of cascaded attention layers is the target action latent variable. Specifically, the feedforward network module is configured to further fuse the first joint feature, the fused action latent variable, and the second joint feature to obtain a fused intermediate action feature; and then sum the fused intermediate action feature with the action latent variable to obtain the output of the attention layer, i.e., the updated action latent variable. The updated action latent variables are used as input for the next attention layer. In this way, after N attention layers process the input features, the target action latent variables are obtained.
[0031] It should be understood that the current action latent variable is the action latent variable corresponding to the attention layer currently in the computation process, and the current action latent variable represents the complete action that the robot will perform next. The current action latent variable input by the first attention layer among multiple attention layers is the initial action latent variable, which is an initial feature vector representing the action intention pre-designed and generated by those skilled in the art when implementing the embodiments of this application. In some embodiments, the initial action latent variable can be generated by random sampling. For example, after determining the dimension of the initial feature vector, a vector of a fixed dimension is randomly sampled from a standard normal distribution or a uniform distribution as the initial action latent variable. In other embodiments, the dimension of the initial action latent variable can be customized by those skilled in the art when implementing the embodiments of this application. For example, a unit vector of a fixed dimension can be set as the initial action latent variable. The dimension of the initial action latent variable can be set by those skilled in the art according to the input dimensions of multiple attention layers, and the dimension of the initial action latent variable must meet the input dimension requirements of multiple attention layers.
[0032] In this embodiment, multiple attention modules are used to process information from different modalities separately, thereby extracting features from various types of information more fully. Independent attention modules are set up for information extraction from different modalities, such as visual features and query features, ensuring that information from each modality can be effectively utilized, avoiding over-reliance on any one modality, and improving the stability of action decisions.
[0033] Step S30: Decode the latent variables of the target action to obtain the action sequence, and control the robot to perform the corresponding actions based on the action sequence.
[0034] In this embodiment, the target motion latent variable is a feature representation characterizing the robot's action intention. Decoding the target motion latent variable restores it to a continuous action sequence. This decoding can employ a linear-layer or multi-layer perceptron structure. Through linear transformation and feature mapping, the high-dimensional abstract target motion latent variable is decoded into a continuous action sequence. The action sequence includes the robot's end effector's initial posture, continuous transition postures during the action, key postures during action execution, and the final posture upon termination of the action. Based on a pre-set total number of action frames and control time intervals, the robot's end effector motion state data for each moment is generated sequentially. In some complex task execution scenarios, each frame of the action sequence fully contains the position information of all controlled joints of the robot and the spatial pose information of the end effector, forming a continuous and time-ordered set of postures.
[0035] In some embodiments, a basic rationality test is performed on the generated continuous motion sequence, the amplitude of posture change between adjacent frames is calculated, abnormal segments with excessive inter-frame abrupt changes, motion speed exceeding the normal range, and posture obviously violating the robot's motion logic are screened and corrected, and the overall motion trend is kept continuous, smooth and in accordance with the laws of physical motion through inter-frame transition adjustment, thus obtaining a corrected motion sequence.
[0036] The dimensions of the motion sequence correspond one-to-one with the actual controllable joints and actuators of the robot. Based on the robot's hardware parameters, the motion sequence is constrained, including: limiting joint positions to safe angle ranges, limiting inter-frame motion speed to the maximum allowable value of the motor, and limiting motion acceleration to avoid impacts and jitter. Smooth interpolation is applied to the changes between frames to make the motion transitions more natural and avoid abrupt jumps. The motion sequence is resampled according to the actual operating frequency of the robot controller to match the number of motion frames and time intervals with the controller cycle. The motion target of each frame is encapsulated into a standard instruction structure recognizable by the controller, including instruction type, target position, running time, priority, etc., forming a complete and quantifiable control instruction sequence. This control instruction sequence is sent to the robot's underlying controller via a communication interface. After receiving the instructions, the controller first performs a validity check on the entire trajectory to confirm the absence of dangerous instructions such as those exceeding the range or time limit. After successful verification, an execution start signal is sent, and the controller begins executing the instructions sequentially. The controller outputs control quantities at fixed intervals to drive the motors or actuators and provides real-time feedback on the current position. Once the last frame of instructions has been executed, the controller returns a completion signal, confirming the end of the action.
[0037] In this embodiment, each attention layer includes multiple attention modules. The first attention module performs cross-modal fusion of action latent variables and query features, the second attention module performs self-fusion of the action latent variables themselves, and the third attention module achieves refined cross-fusion of action features and visual features. The use of multiple attention modules avoids information interference and detail loss caused by a single mixed attention mechanism. The features output by each attention module are further fused through a feedforward network to obtain updated action latent variables. These action latent variables are updated through multiple attention layers to finally obtain the target action latent variables and decode them to generate an action sequence. The aforementioned multiple attention modules can focus on semantic alignment, action cohesion, and key visual information respectively, achieving orderly and efficient fusion of multimodal information, improving the modeling ability of visual details and task objectives, and enhancing the accuracy of action sequence prediction.
[0038] In one feasible implementation, the first attention module is further configured to: obtain a first query vector based on the current action latent variable; obtain a first key vector and a first value vector based on the query features; obtain a first weight matrix based on the first query vector and the first key vector; and perform weighted fusion on the first value vector based on the first weight matrix to obtain a first joint feature. Here, the current action latent variable represents the complete action the robot will perform next; the first query vector is a feature representation of the robot's action query intent, i.e., the information required for the current robot action; the first key vector is a feature representation of the image feature matching index; and the first value vector is a feature representation of the actual content of the image.
[0039] In some embodiments, the input features are linearly transformed using three learnable weight matrices to obtain the query vector, key vector, and value vector corresponding to the input features. The current action latent variable represents the action the robot will perform next, and the query features are input features obtained based on the acquired target image and robot control commands. The first attention module fuses the current action latent variable and the query features. Therefore, during feature fusion, the query vector corresponding to the current action latent variable is used as the query vector for feature fusion, and the key vector and value vector corresponding to the query features are used as the key vector and value vector for feature fusion. The query vector corresponding to the current action latent variable is denoted as the first query vector, and the key vector and value vector corresponding to the query features are denoted as the first key vector and the first value vector, respectively. The first attention module calculates the first weight matrix based on the first query vector and the first key vector; it then performs weighted fusion on the first value vector based on the first weight matrix to obtain the first joint feature. The first weight matrix is obtained by normalizing the dot product similarity between the first query vector and the first key vector. Each element in the first weight matrix represents the correlation between the current action latent variable and the query features. Normalization can be performed using any normalization method, such as Softmax.
[0040] In some embodiments, the second attention module performs self-fusion on the current action latent variable to obtain a fused action latent variable, including: obtaining a second query vector, a second key vector, and a second value vector based on the current action latent variable; obtaining a second weight matrix based on the second query vector and the second key vector; and performing weighted fusion on the second value vector based on the second weight matrix to obtain a self-fused feature. The second weight matrix is obtained by normalizing the dot product similarity between the second query feature and the key of the second key vector.
[0041] In some embodiments, cross-modal fusion of the current action latent variable and visual features includes: obtaining a third query vector based on the current action latent variable; obtaining a third key vector and a third value vector based on the visual features; obtaining a third weight matrix based on the third query vector and the third key vector; determining the top M elements with the highest values in each row of the third weight matrix, and setting the remaining elements to preset masking values to obtain a fourth weight matrix; M is a natural number greater than 0; and performing weighted fusion on the third value vector based on the fourth weight matrix to obtain a second joint feature. In some embodiments, a linear transformation is performed on the current action latent variable based on a preset first activation function to obtain a third query vector, and a linear transformation is performed on the visual features based on a preset second activation function to obtain a third key vector and a third value vector. The first and second activation functions can be the same or different activation functions, and this embodiment is not limited to this. Activation functions include, but are not limited to, ReLU, GELU, SiLU, etc. Multiplying the transposes of the third query vector and the third key vector yields the third weight matrix. The i-th row and j-th column of the third weight matrix represents the correlation between the i-th action latent variable and the j-th visual feature. The sum of each row in the normalized third weight matrix is 1. Traversing the third weight matrix, the M elements with the highest scores in each row are retained; that is, the M visual features most correlated with the action latent variables are retained. All other elements are set to masked values, such as negative infinity or minimum values, thus filtering out irrelevant background or interference information, resulting in the fourth weight matrix. In this embodiment, by calculating the correlation between the action latent variables and visual features, the top M visual regions most relevant to the current task are identified and retained. The feature scores of the remaining irrelevant regions are set to masked values, and after normalization, attention weights are obtained. These attention weights are then weighted and fused to obtain the initial visual attention features. In this way, the third attention module can focus more intently on key visual information, such as the position of the target object or the operation area, thereby improving the accuracy of action prediction.
[0042] In some embodiments, the third value vector is weighted and fused based on the fourth weight matrix to obtain the second joint feature, including: weighting and fusing the third value vector based on the fourth weight matrix to obtain visual attention features; performing linear transformation and nonlinear activation on the current action latent variable to obtain gating coefficients. The corresponding gating coefficients are then multiplied element-wise with the visual attention features to obtain the second joint feature. The gating coefficients are between 0 and 1, representing the correlation strength and confidence level between each visual feature and the current action latent variable. The gating coefficients are weighted element-wise with the visual attention features. For visual features strongly correlated with the task and with high confidence, the gating coefficient is close to 1, achieving the preservation and appropriate enhancement of the corresponding visual information; for weakly correlated, noisy, or interfering features, the gating coefficient approaches 0, achieving the suppression and filtering of the corresponding visual information. The visual features purified by the gating mechanism eliminate redundant and interfering information, retaining only key visual content with high confidence and close relevance to the task, providing cleaner, more focused, and more discriminative input features for the subsequent strategy decision-making module, thereby effectively improving the accuracy and robustness of action prediction.
[0043] In some embodiments, the feedforward network module obtains updated action latent variables based on the first joint feature, the fused action latent variable, and the second joint feature, including: concatenating the first joint feature, the fused action latent variable, and the second joint feature to obtain concatenated features; performing a linear transformation and nonlinear activation on the concatenated features to obtain refined features; and adding the refined features and the current action latent variable element-wise to obtain updated action latent variables. In some embodiments, the concatenated features are input into a multi-layer fully connected neural network to obtain refined features. Specifically, the multi-layer fully connected neural network performs a first linear transformation on the concatenated features, using learnable weights to increase the dimensionality of the concatenated features, obtaining a high-dimensional intermediate feature. This allows the multi-layer attention layers to mine subtle differences between features in a higher-dimensional space, extracting more fine-grained key information, thereby providing a richer feature space for subsequent nonlinear transformations. An activation function is applied to the intermediate features output by the first linear transformation to perform a nonlinear transformation, allowing the model to learn more complex interaction relationships between the three types of information. The activation function can be any of the nonlinear activation functions such as GELU or ReLU, and this embodiment is not limited to this. The activated intermediate features are input into the second linear layer. Learnable weights are used to map the feature dimensions from the intermediate dimensions back to dimensions completely consistent with the original action query features, resulting in fused features that match the input dimensions. The refined features are then added element-wise to the current action latent variables to obtain updated action latent variables. The updated action latent variables retain the core information already learned from the original action query features, preventing the loss of key intents during multiple transformations and avoiding the gradient vanishing problem.
[0044] In one feasible implementation, the target image includes a global image and a wrist-specific local image. Encoding the acquired target image to obtain visual features includes: extracting features from the acquired global image and wrist image respectively to obtain global visual features and local visual features; and stitching the global and local visual features together to obtain the final visual features. The global image includes the overall environmental spatial layout, the absolute position of the object to be manipulated in the scene, the global pose of the robot body and end effector in the scene, and the distribution of operable and obstacle areas within the scene, providing underlying visual support for the robot to plan its global motion path and determine the workspace range. The wrist image is a local image including the target object and the end effector. The wrist image can compensate for the insufficient representation of local details in the global image, improving the accuracy of the visual features.
[0045] Another embodiment of this application provides a robot control device, the device comprising: The acquisition module is configured to process the acquired target image and robot control commands to obtain visual features and query features; An attention module is configured to derive a target action latent variable based on the input visual features, the query features, and an initial action latent variable. The attention module comprises multiple cascaded attention layers, each of which includes a first attention module, a second attention module, a third attention module, and a feedforward network module. Specifically, the first attention module is configured to perform cross-modal fusion on the current action latent variable and the query features to obtain a first joint feature; the second attention module is configured to perform self-fusion on the current action latent variable to obtain a fused action latent variable; the third attention module is configured to perform cross-modal fusion on the current action latent variable and the visual features to obtain a second joint feature; the feedforward network module is configured to obtain an updated action latent variable based on the first joint feature, the fused action latent variable, and the second joint feature; the current action latent variable is the updated action latent variable obtained after at least one update of the initial action latent variable; the current action latent variable corresponding to the first attention layer in the multiple cascaded attention layers is the initial action latent variable; and the updated action latent variable obtained by the last attention layer in the multiple cascaded attention layers is the target action latent variable. The decoding module is configured to decode the latent variables of the target action to obtain an action sequence, and control the robot to perform the corresponding action based on the action sequence.
[0046] Each module described above corresponds to a step in each method embodiment, and the function of each module is determined based on the steps in each method embodiment. Please refer to the descriptions of the method embodiments described above; they will not be repeated here.
[0047] Another embodiment of this application provides a robot control device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the robot control method in any of the above-described method embodiments.
[0048] The following is for reference. Figure 4 The diagram illustrates a structural schematic of a robot control device suitable for implementing the robot control method of the embodiments of this application. The robot control device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, personal digital assistants (PDAs), tablet computers (PADs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and main positioning terminals such as digital TVs and desktop computers. Figure 4 The robot control device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this application.
[0049] like Figure 4 As shown, the robot control device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that executes various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the robot control device. The processing unit 1001, the ROM 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the robot control device to communicate wirelessly or wiredly with other devices to exchange data.
[0050] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, the steps defined in the method embodiments disclosed in this application are performed.
[0051] The robot control device provided in this application, when executing the robot control method in the above embodiments, can solve the technical problem of inaccurate action sequence prediction. Compared with the prior art, the beneficial effects of the robot control device provided in this application are the same as those of the robot control method provided in the above embodiments, and will not be repeated here.
[0052] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0053] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0054] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the robot control method in the above embodiments.
[0055] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.
[0056] The aforementioned computer-readable storage medium may be included in the robot control device or may exist independently. The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the robot control device, the robot control device performs the various steps corresponding to the robot control method. Please refer to the various steps of the aforementioned robot control method; they will not be elaborated upon here.
[0057] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the client computer, partially on the client computer, as a standalone software package, partially on the client computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the client computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0058] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of the systems, methods, and computer program products of various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0059] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.
[0060] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described robot control method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the robot control method provided in the above embodiments, and will not be repeated here.
[0061] One embodiment of this application provides a robot control system, which includes a robot and a robot control device. The robot is equipped with a camera for acquiring target images. The robot control device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the steps of any of the robot control method embodiments described above. Compared with the prior art, the beneficial effects of the robot control system provided in this application are the same as those of the robot control methods provided in the above embodiments, and will not be repeated here.
[0062] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.
Claims
1. A robot control method, characterized in that, The method includes: The acquired target image and robot control commands are processed to obtain visual features and query features; The visual features, the query features, and the initial action latent variable are input into multiple cascaded attention layers to obtain the target action latent variable. Each of the multiple cascaded attention layers includes a first attention module, a second attention module, a third attention module, and a feedforward network module. The first attention module is configured to perform cross-modal fusion on the current action latent variable and the query features to obtain a first joint feature. The second attention module is configured to perform self-fusion on the current action latent variable to obtain a fused action latent variable. The third attention module is configured to perform cross-modal fusion on the current action latent variable and the visual features to obtain a second joint feature. The feedforward network module is configured to obtain an updated action latent variable based on the first joint feature, the fused action latent variable, and the second joint feature. The current action latent variable is the updated action latent variable obtained after at least one update of the initial action latent variable. The current action latent variable corresponding to the first attention layer in the multiple cascaded attention layers is the initial action latent variable. The updated action latent variable obtained by the last attention layer in the multiple cascaded attention layers is the target action latent variable. The latent variables of the target action are decoded to obtain the action sequence, and the robot is controlled to perform the corresponding action based on the action sequence.
2. The robot control method as described in claim 1, characterized in that, The cross-modal fusion of the current action latent variable and the visual features includes: A third query vector is obtained based on the current action latent variables; a third key vector and a third value vector are obtained based on the visual features; The third weight matrix is obtained based on the third query vector and the third key vector; The top M elements with the highest values in each row of the third weight matrix are determined, and the remaining elements are set to preset masking values to obtain the fourth weight matrix; where M is a natural number greater than 0. The second joint feature is obtained by weighting and fusing the third value vector based on the fourth weight matrix.
3. The robot control method as described in claim 2, characterized in that, The weighted fusion of the third value vector based on the fourth weight matrix to obtain the second joint feature includes: The third value vector is weighted and fused based on the fourth weight matrix to obtain visual attention features; The current action latent variable is subjected to linear transformation and nonlinear activation to obtain the gating coefficients; The corresponding gating coefficient is multiplied element-wise with the visual attention feature to obtain the second joint feature.
4. The robot control method as described in claim 1, characterized in that, The updated action latent variable obtained based on the first joint feature, the fused action latent variable, and the second joint feature includes: The first joint feature, the fused action latent variable, and the second joint feature are concatenated to obtain the concatenated feature; The spliced features are subjected to linear transformation and nonlinear activation to obtain refined features; The refined features and the current action latent variables are added element by element to obtain the updated action latent variables.
5. The robot control method as described in claim 1, characterized in that, The process of processing the acquired target image and robot control commands to obtain visual features and query features includes: The acquired target image is encoded to obtain visual features; The robot control commands are processed to obtain command characteristics; The visual features and the instruction features are fused to obtain the query features.
6. The robot control method as described in claim 5, characterized in that, The target image includes a global image and a local image of the wrist. Encoding the acquired target image to obtain visual features includes: Feature extraction is performed on the acquired global image and wrist image respectively to obtain global visual features and local visual features; The visual features are obtained by concatenating the global visual features and the local visual features.
7. A robot control device, characterized in that, The device includes: The acquisition module is configured to process the acquired target image and robot control commands to obtain visual features and query features; An attention module is configured to derive a target action latent variable based on the input visual features, the query features, and an initial action latent variable. The attention module comprises multiple cascaded attention layers, each of which includes a first attention module, a second attention module, a third attention module, and a feedforward network module. Specifically, the first attention module is configured to perform cross-modal fusion on the current action latent variable and the query features to obtain a first joint feature; the second attention module is configured to perform self-fusion on the current action latent variable to obtain a fused action latent variable; the third attention module is configured to perform cross-modal fusion on the current action latent variable and the visual features to obtain a second joint feature; the feedforward network module is configured to obtain an updated action latent variable based on the first joint feature, the fused action latent variable, and the second joint feature; the current action latent variable is the updated action latent variable obtained after at least one update of the initial action latent variable; the current action latent variable corresponding to the first attention layer in the multiple cascaded attention layers is the initial action latent variable; and the updated action latent variable obtained by the last attention layer in the multiple cascaded attention layers is the target action latent variable. The decoding module is configured to decode the latent variables of the target action to obtain an action sequence, and control the robot to perform the corresponding action based on the action sequence.
8. A robot control device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the robot control method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The storage medium is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the robot control method as described in any one of claims 1 to 6.
10. A robot control system, characterized in that, The robot control system includes a robot and a robot control device; the robot is equipped with a camera for acquiring target images; the robot control device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the robot control method as described in any one of claims 1 to 6.