Robot operation task control method, device and equipment based on visual language action model, robot and medium

Abstract

The invention provides a robot operation task control method, device and equipment based on a visual language action model, a robot and a medium, and relates to the technical field of sensors and robots. The method comprises the following steps: predicting weights between visual angles of corresponding visual image data based on global feature vectors corresponding to the visual image data of a plurality of visual angles of a robot; according to the plurality of local feature vectors corresponding to each piece of visual image data, predicting the weight in the visual angle of the corresponding image area; fusing the feature vectors of the reserved areas in the visual images in combination with the inter-view-angle weight and the in-view-angle weight of each image area to obtain a fused feature vector; and inputting the fusion feature vector into a robot operation task control strategy network based on a visual language action model to obtain an operation task control instruction. The motion execution precision can be improved, the computing power burden is reduced, the computing efficiency is improved, and the task success rate of complex and fine tasks is improved.

Description

Technical Field

[0001] This disclosure relates to the fields of sensor and robotics technology, and to a method, apparatus, device, robot, and medium for controlling robot operation tasks based on a visual language action model. Background Technology

[0002] Current robots often employ multi-view visual perception strategies when performing fine manipulation tasks (such as grasping, inserting, and aligning). However, existing technologies have the following problems: Redundancy of feature information: Some existing solutions use simple splicing or equal weight fusion strategies, which fail to effectively compress or filter information, resulting in the introduction of a large number of useless or duplicate image features during the operation, which significantly increases the computational burden.

[0003] Insufficient perspective adaptation capability: Some existing technologies cannot dynamically determine the effectiveness of each perspective based on the progress of the task. They usually adopt fixed perspective combinations or static fusion rules, and lack a selection mechanism for perspectives at each stage.

[0004] Limited policy accuracy: Due to the failure to fully highlight key perspective information during the feature fusion stage, there is an information dilution problem in the fused features, resulting in low accuracy of the policy network when generating actions, especially under complex tasks.

[0005] The aforementioned problems have been addressed in the paper "BFA: Best-Feature-Aware Fusion for Multi-View Fine-grained Manipulation" (ZihanLan et al., June 28, 2025). However, the inventors have discovered the following issues in practice: First, BFA only assigns viewpoint weights at the overall image level and does not distinguish the importance of regions within the image. As a result, the model retains a large number of background features that are irrelevant to the task during processing, which interferes with subsequent feature fusion and action generation.

[0006] Second, the weight prediction network in BFA can only reflect the relative importance between different viewpoints and lacks a mechanism for identifying key regions within a single image. For some operational stages, the overall viewpoint of some images is not significant, but local regions (such as the end effector of the robot arm or the target contact area) may be crucial to task completion. BFA cannot accurately identify and enhance such local features.

[0007] All of these factors can affect the success rate and accuracy of complex and sophisticated tasks. Summary of the Invention

[0008] This disclosure provides a robot operation task control method, device, equipment, robot, and medium based on a visual language action model, which can eliminate a large number of background features that are irrelevant to the task, identify and enhance the local features of key areas, and improve the success rate and execution accuracy of complex and delicate tasks.

[0009] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure.

[0010] According to a first aspect of this disclosure, a robot operation task control method based on a visual language action model is provided, the method comprising: Based on the global feature vectors corresponding to the visual image data from multiple perspectives of the robot, a pre-trained inter-view importance prediction network is used to predict the inter-view weights of the corresponding visual image data. For each image region in each visual image data, feature extraction is performed to obtain multiple local feature vectors corresponding to that visual image data. Based on local feature vectors, a pre-trained in-view importance prediction network is used to predict the in-view weights of the corresponding image regions. By combining the interview weights and the intraview weights of each image region, the feature vectors of the preserved regions in each visual image data are fused to obtain the fused feature vector. The fused feature vectors are input into a robot operation task control strategy network based on a visual language action model to obtain operation task control instructions.

[0011] In one exemplary embodiment of this disclosure, the feature vectors of the preserved regions in each visual image data are fused by combining the inter-viewpoint weights and the intra-viewpoint weights of each image region to obtain a fused feature vector, including: Based on the interview weights and intraview weights, the image regions in each visual image data are pruned to obtain the retained regions in each visual image data. The feature vectors of the preserved regions in each visual image data are fused to obtain a fused feature vector.

[0012] In one exemplary embodiment of this disclosure, the feature vectors of the preserved regions in each visual image data are fused by combining the inter-viewpoint weights and the intra-viewpoint weights of each image region to obtain a fused feature vector, including: The global importance score of each image region is obtained by multiplying the interview weights of each visual image data with the intraview weights of the corresponding image regions. Image regions with a global importance score greater than a preset score threshold are identified as reserved regions; The local feature vectors of each retained region are weighted according to the global importance score to obtain the weighted feature vectors of each retained region; The weighted feature vectors of each preserved region in each visual image data are concatenated to obtain the fused feature vector.

[0013] In one exemplary embodiment of this disclosure, image regions in each visual image data are pruned based on inter-viewpoint weights and intra-viewpoint weights to obtain retained regions in each visual image data, including: Obtain the interview weights corresponding to each visual image data, and determine the visual image data whose interview weights are greater than the preset interview threshold as candidate view data. The image regions in each candidate viewpoint data are sorted based on their corresponding viewpoint weights to obtain the sorting results; Based on the sorting results, the reserved regions in each candidate viewpoint data are determined, and the image regions of the reserved regions are selected from each candidate viewpoint data.

[0014] In one exemplary embodiment of this disclosure, image regions in each visual image data are pruned based on inter-viewpoint weights and intra-viewpoint weights to obtain retained regions in each visual image data, including: By utilizing the in-view weights of image regions in each visual image data, local pruning is performed on each image region to obtain the primary preserved region of each visual image data. The global importance score of each image region in each primary preservation region is calculated based on the inter-viewpoint weights and intra-viewpoint weights. Based on the global importance score of each image region in each primary preserved region, global pruning is performed on each primary preserved region in all visual image data to obtain the preserved regions in each visual image data.

[0015] In one exemplary embodiment of this disclosure, based on local feature vectors, a pre-trained in-view importance prediction network is used to predict the in-view weights of corresponding image regions, including: An in-view importance prediction network is used to initially score local feature vectors, resulting in the original importance score of the image region to be predicted. Calculate the spatial distance between the image region to be predicted and other image regions in the visual image data; where other image regions are all image regions in the visual image data other than the image region to be predicted. The original importance scores are smoothed and weighted using spatial distance to obtain the processed importance scores; The importance scores of each image region in the visual image data are normalized to obtain the in-view weight of the image region to be predicted.

[0016] In one exemplary embodiment of this disclosure, the original importance score is smoothed and weighted using spatial distance to obtain a processed importance score, including: The ratio of the original importance score of other image regions to their corresponding spatial distance is determined as the distance attenuation term for other image regions; The correction value is determined by the weighted sum between the original importance scores of other image regions and the distance attenuation term; The sum of the corrected value and the original importance score is determined as the processed importance score.

[0017] In one exemplary embodiment of this disclosure, based on each global feature vector, a pre-trained inter-view importance prediction network is used to predict the inter-view weights of the corresponding visual image data, including: The global feature vectors of each visual image data are concatenated to construct a multi-view context vector; The multi-view context vector is input into the inter-view importance prediction network, and the initial prediction value is output through the inter-view importance prediction network. The initial predicted values ​​are normalized to obtain the weights between the viewpoints of each visual image data.

[0018] In one exemplary embodiment of this disclosure, feature extraction is performed on each image region in each visual image data to obtain multiple local feature vectors corresponding to the visual image data, including: The sequence of each image region in each visual image data is input into a pre-trained visual encoder. The self-attention mechanism of the pre-trained visual encoder is used to process each image region to generate a feature sequence including category tokens and image block tokens. Image block tokens are extracted from the feature sequence, and local feature vectors corresponding to the visual image data are determined based on the image block tokens corresponding to each image region.

[0019] According to a second aspect of this disclosure, a method for training a robot operation task control model based on a visual language action model is provided, the method comprising: The ground truth labels for importance between viewpoints and importance within viewpoints are determined based on the training data. The training data includes visual image data from multiple viewpoints during the robot's task execution, robot action labels corresponding to each visual image data, and robot state description information. The inter-view importance prediction network is used to determine the inter-view prediction weights based on the global feature vectors of the visual image data, and the intra-view importance prediction network is used to determine the intra-view prediction weights based on the local feature vectors of the visual image data. The local feature vectors are weighted and fused by combining the prediction weights between and within the viewpoints, and the predicted action is obtained by using the robot operation task control strategy network based on the fused feature vector. Based on the first auxiliary loss between the inter-view prediction weights and the ground truth labels of inter-view importance, the second auxiliary loss between the intra-view prediction weights and the ground truth labels of intra-view importance, and the action prediction loss between the predicted action and the robot action label, the parameters of the inter-view importance prediction network, the intra-view importance prediction network, and the policy network are updated respectively.

[0020] In one exemplary embodiment of this disclosure, determining ground truth labels for inter-view importance and intra-view importance based on training data includes: Based on the robot state description information corresponding to each visual image data, determine the ground truth labels of the importance between the viewpoints of each visual image data. Based on the correspondence between each image region and the task entity in each visual image data, generate ground truth labels for the visual image data within the viewpoint.

[0021] In one exemplary embodiment of this disclosure, determining the ground truth labels of the importance between viewpoints of each visual image data based on the robot state description information corresponding to each visual image data includes: Determine the current operational stage of each robot based on its status description information; If the operation phase is the object interaction phase, then the viewpoint importance truth labels of the visual image data corresponding to the main viewpoint and the visual image data containing at least one robot arm viewpoint in each visual image data are marked as the first value. If the operation phase is a non-interactive phase, then the ground truth label of the importance between viewpoints of the visual image data corresponding to the main viewpoint in each visual image data is marked as the first value, and the ground truth label of the importance of the visual image data containing the robot arm's viewpoint is marked as the second value.

[0022] In one exemplary embodiment of this disclosure, a ground truth label for the visual importance of each visual image data is generated based on the correspondence between each image region and the task entity in each visual image data, including: Obtain the target region range of the task entity in each visual image data; In each visual image data, the pixel that is located within the corresponding target region is determined as the first pixel, and the importance truth label of the first pixel within the viewpoint is marked as the first value. In each visual image data, pixels that are not located within the corresponding target area are identified as second pixels, and the ground truth label of the importance of the second pixel within the viewpoint is marked as the second value.

[0023] In one exemplary embodiment of this disclosure, the task entity includes at least a robotic arm actuator and an object being manipulated.

[0024] According to a third aspect of this disclosure, a robot operation task control device based on a visual language action model is provided, the device comprising: The first prediction module is used to predict the inter-view weights of the corresponding visual image data based on the global feature vectors corresponding to the visual image data from multiple perspectives of the robot, using a pre-trained inter-view importance prediction network. The extraction module is used to extract features from each image region in each visual image data to obtain multiple local feature vectors corresponding to the visual image data. The second prediction module is used to predict the view weights of the corresponding image regions based on local feature vectors and using a pre-trained view importance prediction network. The first fusion module is used to combine the interview weights and the intraview weights of each image region to fuse the feature vectors of the preserved regions in each visual image data to obtain a fused feature vector. The strategy output module is used to input the fused feature vector into the robot operation task control model pre-trained based on the visual language action model algorithm to obtain operation task control instructions.

[0025] According to a fourth aspect of this disclosure, a training device for a robot operation task control model based on a visual language action model is provided, the device comprising: The determination module is used to determine the ground truth labels of importance between viewpoints and importance within viewpoints based on the training data; wherein, the training data includes visual image data from multiple viewpoints during the robot's task execution, robot action labels corresponding to each visual image data, and robot state description information; The third prediction module is used to determine the prediction weights between viewpoints based on the global feature vector using the inter-view importance prediction network, and to determine the prediction weights within the viewpoint based on the local feature vector using the intra-view importance prediction network. The second fusion module is used to combine the inter-view prediction weights and intra-view prediction weights to perform weighted fusion of local feature vectors, and to obtain the predicted action based on the fused feature vector. The policy training module is used to update the parameters of the inter-view importance prediction network, the intra-view importance prediction network, and the policy network based on the first auxiliary loss between the inter-view prediction weights and the ground truth labels of inter-view importance, the second auxiliary loss between the intra-view prediction weights and the ground truth labels of intra-view importance, and the action prediction loss between the predicted action and the robot action label.

[0026] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: Processor; and A memory that stores computer-readable instructions, which, when executed by a processor, implement the method as described in the above embodiments.

[0027] According to a sixth aspect of this disclosure, a robot is provided, comprising: Processor; and A memory that stores computer-readable instructions, which, when executed by a processor, implement the method as described in the above embodiments.

[0028] In one exemplary embodiment of this disclosure, the robot includes either a humanoid robot or a dual-arm robot.

[0029] According to a seventh aspect of this disclosure, a computer-readable storage medium is provided that stores computer program code instructions, which, when invoked by a robot's processor, cause the robot to perform the method as described in the above embodiments.

[0030] As can be seen from the above technical solution, this disclosure possesses at least one of the following advantages and positive effects: This disclosure, based on the prediction of the interview weights of the entire image in visual image data, further obtains the intraview weights of the internal image regions of the visual image data. Then, based on the interview weights and intraview weights, it performs feature fusion on the retained regions in the visual image data. This process eliminates task-irrelevant background features, reducing interference from irrelevant features on subsequent feature fusion and action generation, while also reducing computational burden and improving computational efficiency. By determining the operation task control instructions based on the fused feature vectors of the retained regions, it ensures that even in complex and fine-grained operation tasks, the local features of key regions can be transmitted to the operation task control policy network, even if the overall viewpoint of the image is not significant. This achieves the identification and enhancement of local features in key regions, avoiding the dilution of key fine-grained features and improving the success rate and accuracy of complex and fine-grained tasks. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1A system architecture diagram is shown for a robot operation task control method based on a visual language action model that can be applied to embodiments of this disclosure.

[0033] Figure 2 A flowchart illustrating a robot operation task control method based on a visual language action model that can be applied to embodiments of this disclosure is shown.

[0034] Figure 3 A flowchart illustrating a method for determining fused feature vectors that can be applied to embodiments of this disclosure is shown.

[0035] Figure 4 A schematic diagram is shown that can be applied to the pruning process in the embodiments of this disclosure.

[0036] Figure 5 A flowchart illustrating a method for determining in-view weights that can be applied to embodiments of this disclosure is shown.

[0037] Figure 6 A flowchart illustrating a training method for a robot operation task control model based on a visual language action model, which can be applied to embodiments of this disclosure, is shown.

[0038] Figure 7 A schematic diagram of the process for generating importance truth labels that can be applied to embodiments of this disclosure is shown.

[0039] Figure 8 A schematic diagram of the operational architecture of a robot operation task control model based on a visual language action model, which can be applied to the embodiments of this disclosure, is shown.

[0040] Figure 9 A visual schematic diagram illustrating the visual token pruning effect that can be applied to embodiments of this disclosure is shown.

[0041] Figure 10 A schematic diagram illustrating the dynamic change process of the weights between different viewpoints over time, which can be applied to the embodiments of this disclosure, is shown.

[0042] Figure 11 The effect of pruning ratios applicable to embodiments of this disclosure on the success rate of robot tasks is illustrated.

[0043] Figure 12 A block diagram of a robot operation task control device based on a visual language action model is shown in an embodiment of the present disclosure.

[0044] Figure 13 A block diagram of a training device for a robot operation task control model based on a visual language action model is shown in an embodiment of the present disclosure.

[0045] Figure 14 A schematic diagram of a humanoid robot according to an embodiment of this disclosure is shown.

[0046] Figure 15 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.

[0047] Figure 16 A schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0048] In this disclosure, the terms "first" and "second" are used for description only and do not indicate relative importance or imply the number of technical features. Therefore, the features referred to as "first" or "second" may explicitly or implicitly include at least one of those features. "A plurality of" means at least two, unless otherwise expressly defined.

[0049] First, the relevant terms used in the exemplary embodiments of this disclosure will be explained: Vision-Language-Action Model (VLA): This is used to represent an end-to-end deep learning model that can process visual information and natural language instructions simultaneously.

[0050] Token: Used to represent the basic unit for processing data in the encoder architecture. In computer vision (Vision Transformer), it usually refers to the feature vector generated by dividing the input image into multiple fixed-size image patches, followed by linear projection and positional encoding.

[0051] Inter-view Importance: A metric used to represent the contribution of a particular camera viewpoint to the overall current task in a multi-camera viewpoint system.

[0052] Intra-view Importance: A metric used to measure the degree of attention given to different image regions relative to the current task within an image viewed from a single camera.

[0053] Attention Mechanism: This refers to a mechanism in neural networks that simulates human visual focus. It assigns weights by calculating the similarity between the query vector and the key vector, enabling the model to focus on key information in the input sequence.

[0054] Ground Truth (GT): In supervised learning, it refers to the true result or standard answer corresponding to the sample. It is used to calculate the error (Loss) of the model output and guide parameter updates.

[0055] End-effector: refers to a device installed at the end of a robot arm that directly interacts with the environment physically, such as a gripper, suction cup, or dexterous hand.

[0056] Figure 1 A system architecture diagram is shown that can be applied to the robot operation task control method based on visual language action model in the embodiments of this disclosure.

[0057] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a robot 102, a network 103, and a server 104.

[0058] The terminal device 101 includes, but is not limited to, desktop computers, laptops, smartphones, and tablets. The terminal device 101 is equipped with a graphical user interface (GUI), providing human-computer interaction capabilities. It allows users to input operation commands in natural language and displays multi-view monitoring images and task execution status of the robot 102 in real time. The GUI can also serve as an interactive control interface, supporting users to implement robot motion control processes as needed. Furthermore, the terminal device 101 provides model configuration functions, allowing users to set strategy model types, debug training processes, etc., and collaborate with the server 104 to complete the visualization analysis and management of training data.

[0059] Robot 102 is equipped with multiple vision acquisition devices for real-time acquisition of multi-view visual image data covering the work area. Robot 102 includes at least a main controller and actuators. The main controller is equipped with a high-performance computing unit for running the deployed deep learning model architecture. The actuators, such as joint actuators of the robot arm and end effector actuators of the robot arm, receive motion control commands from the main controller and drive the robot arm and end effector to perform corresponding fine operations, thereby enabling the robot to adaptively control complex environments.

[0060] Server 104 is equipped with a large-scale deep learning training platform that can receive and preprocess massive amounts of multimodal data from robot 102 or public datasets, including images, language text, and motion trajectories. This data is used to pre-train or fine-tune the inter-view importance prediction network, the intra-view importance prediction network, and the visual-language-action model. After training, server 104 can distribute and deploy the optimized model parameters or a lightweight inference model compressed package to the edge computing end of robot 102 via network 103, supporting its real-time inference and decision-making in offline or online states.

[0061] Network 103 serves as a medium for providing a communication link between terminal device 101, robot 102, and server 104. Network 103 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to ensure low-latency transmission of multi-view high-definition video streams and control commands.

[0062] Figure 1 The number and type of terminal devices, robots, networks, and servers shown are merely illustrative. Depending on implementation needs, any number and type of terminal devices, robots, networks, and servers can be included.

[0063] Through the coordinated operation of various parts in the system architecture 100, a complete process can be achieved, from multi-view visual information perception, prediction and fusion of intra-view weights and inter-view weights, to the generation of high-precision action strategies from visual language action models, thereby improving the success rate, efficiency and accuracy of robots performing fine operation tasks in unstructured environments.

[0064] This disclosure provides an exemplary implementation of a robot operation task control method based on a visual language action model, referencing... Figure 2 As shown, the method may include the following steps S201 to S205: Step S201: Based on the global feature vectors corresponding to the visual image data from multiple perspectives of the robot, a pre-trained inter-view importance prediction network is used to predict the inter-view weights of the corresponding visual image data.

[0065] Step S202: For each image region in each visual image data, feature extraction is performed to obtain multiple local feature vectors corresponding to the visual image data.

[0066] Step S203: Based on local feature vectors, use a pre-trained in-view importance prediction network to predict the in-view weights of the corresponding image regions.

[0067] Step S204: Combine the interview weights and the intraview weights of each image region to fuse the feature vectors of the preserved regions in each visual image data to obtain a fused feature vector.

[0068] Step S205: Input the fused feature vector into the robot operation task control strategy network based on the visual language action model to obtain the operation task control command.

[0069] The robot operation task control method based on a visual language action model provided in the exemplary embodiments of this disclosure quantifies the influence of different perspectives and regions within the same perspective on the current task from both global and local dimensions by using inter-perspective weights and intra-perspective weights of visual image data from multiple perspectives. This avoids equal-weighting of redundant perspectives, reduces information noise input during feature fusion, and effectively identifies key preserved regions in the visual image data. Furthermore, by generating fusion feature vectors for the preserved regions, background interference can be effectively eliminated, and the preserved regions in the image can be further enhanced. This avoids feature dilution of key regions during feature fusion, reduces computational burden, improves computational efficiency, and enhances the success rate and accuracy of complex and sophisticated tasks.

[0070] The robot operation task control method based on visual language action model in this example embodiment will be described in detail below.

[0071] In step S201, based on the global feature vectors corresponding to the visual image data from multiple perspectives of the robot, a pre-trained inter-view importance prediction network is used to predict the inter-view weights of the corresponding visual image data.

[0072] In some example implementations, during the execution of operational tasks, the robot can simultaneously acquire visual image data of the current environment using multiple visual sensors configured at different spatial locations. The positions of each camera can be pre-fixed or possess a certain degree of mechanical rotational freedom, thereby acquiring visual image data from multiple perspectives. This visual image data can include: RGB (Red, Green, Blue) image data, depth image data, camera pose information, etc. Specifically, the RGB image data is acquired by multiple standard visible light cameras deployed around the robot, providing information about the color, texture, outline, and spatial layout of target objects in the operational environment; the depth image data is generated by structured light cameras, Time-of-Flight (ToF) sensors, or a binocular stereo vision system, used to acquire pixel-level three-dimensional distance information corresponding one-to-one with the RGB images; the camera pose information includes the intrinsic and extrinsic parameter matrices of the cameras mounted on the robot body or robot arm, used to indicate the spatial position and orientation information of each camera in the global coordinate system. This spatial position and orientation information can be used as additional metadata of the image, input into the view importance prediction network, so that the network can fully consider the spatial distribution characteristics of each viewpoint when determining viewpoint weights. By acquiring the above-mentioned multi-type image data and pose data, feature vectors that reflect the observation features from different perspectives and the spatial distribution information of different pixels can be constructed, providing a data foundation for subsequent weight prediction and feature fusion.

[0073] For example, visual image data collected by various visual sensors of the robot can be acquired, and the visual image data can be input into a pre-trained visual encoder to extract the global feature vector corresponding to each visual image. The global feature vector is then input into a pre-trained view importance prediction network to obtain the view weights corresponding one-to-one with each visual image. The visual encoder can be a residual network or the image encoding end of a contrastive language-image pre-trained model; the view importance prediction network can be a shallow fully connected structure, or it can adopt an attention mechanism or a multilayer perceptron structure. This disclosure does not impose any special limitations on the specific network structure.

[0074] The global feature vectors corresponding to all viewpoints can be concatenated to construct a context vector containing multi-view information. This context vector is then input into a pre-trained inter-view importance prediction network. Based on prior knowledge learned during training, this network analyzes the matching relationship between the current robot's action stage and the content within each viewpoint, outputting the original score value for each viewpoint.

[0075] Furthermore, to obtain standardized weights, activation functions, such as the sigmoid function, can be used to map the original score values ​​to a preset range, such as 0 to 1, thus obtaining the inter-viewpoint weights for each visual image. A larger inter-viewpoint weight indicates that the visual information provided by that viewpoint at the current moment is more crucial; a smaller inter-viewpoint weight indicates that the viewpoint may be occluded or contain invalid background at the current moment.

[0076] By using the above methods, the importance of global feature vectors from various perspectives can be predicted, the best perspective can be identified, noise interference from irrelevant perspectives can be suppressed, and the success rate of robot operation tasks in complex environments can be improved.

[0077] In step S202, feature extraction is performed on each image region in each visual image data to obtain multiple local feature vectors corresponding to the visual image data.

[0078] In some example implementations, each input visual image can be segmented into blocks. The entire image can be spatially divided into multiple non-overlapping image blocks according to a preset block size. The location embedding vector for each image block can be determined, and each block can be flattened into a one-dimensional vector. This vector is then mapped to a preset feature dimension using a learnable linear mapping layer. The feature vector corresponding to each image block is element-wise added to the location embedding vector to obtain an initial feature sequence containing location information. This initial feature sequence can be input into a pre-trained visual encoder, which outputs a feature sequence containing multiple vectors. Local feature vectors are then extracted from this feature sequence. This feature sequence includes a category feature for global representation and multiple image block features for local representation. The category feature for global classification can be removed from the feature sequence, and the remaining features corresponding to each image block are determined as multiple local feature vectors for that visual image. Each local feature vector uniquely corresponds to an image region in the original image.

[0079] The above method enables the generation of multiple local feature vectors to fully preserve the information of the visual image data without losing spatial structure information, providing an accurate data foundation for subsequent use of in-view importance prediction networks to obtain the preserved regions inside the image.

[0080] In step S203, based on local feature vectors, a pre-trained in-view importance prediction network is used to predict the in-view weights of the corresponding image regions.

[0081] In some example implementations, each local feature vector can be independently input into a pre-trained in-view importance prediction network. This network analyzes whether the content contained in the local region belongs to a task-related entity, such as a robot arm or a manipulated object, or to irrelevant background noise, such as desktop textures or clutter. The in-view importance network outputs a corresponding scalar score, which is then processed by an activation function to obtain in-view weights. The higher the in-view weight, the more important the task information contained in the image region. For example, the in-view weights can be calculated using the following formula: in, This represents the in-view weight of the nth image region in the visual image data of the vth viewpoint, as output by the in-view importance prediction network. This represents the in-view importance prediction network. This represents the local feature vector corresponding to the image region.

[0082] Since the weights within this viewpoint reflect the image regions in the visual image data corresponding to the current viewpoint that are more important to the completion of the robot's task, such as the target grasping point, the robot arm actuator, and the contact point that interacts with the target, it is possible to effectively distinguish between useful regions and interfering background regions in the visual image data of each viewpoint.

[0083] In step S204, the feature vectors of the preserved regions in each visual image data are fused by combining the interview weights and the intraview weights of each image region to obtain a fused feature vector.

[0084] In some example implementations, a global importance score for each image region can be calculated based on a pre-trained in-view importance prediction network.

[0085] As an example, the inter-viewpoint weights of the viewpoints corresponding to the image region are multiplied by the intra-viewpoint weights of the image region itself to obtain the global importance score of the image region. This score is used to determine the importance of the image region within that viewpoint.

[0086] Pruning and filtering can be performed on each image region based on its global importance score. All image regions from all viewpoints can be sorted according to their global importance score, or compared with a preset threshold. Image regions whose scores meet preset conditions, such as a top-ranking retention ratio or a score greater than a preset importance threshold, are identified as retained regions. Regions that do not meet the conditions, i.e., redundant backgrounds or regions with invalid viewpoints, have their corresponding feature vectors removed.

[0087] The local feature vectors of all preserved regions can be concatenated or stacked to obtain a fused feature vector.

[0088] By defining the regions to be retained, the sequence length of the input data can be reduced. While preserving key image regions, the computational burden on subsequent networks is reduced, computational efficiency is improved, and interference from redundant data is effectively eliminated.

[0089] In step S205, the fused feature vector is input into the robot operation task control strategy network based on the visual language action model to obtain the operation task control command.

[0090] In some example implementations, the robot operation task control policy network can be constructed based on Transformer, RNN (Recurrent Neural Network), or a multimodal fusion model. Since the input fused feature vector has removed a large amount of irrelevant background and occluded viewpoint features, the robot operation task control policy network can directly focus on the preserved region corresponding to the core operation and output predicted operation task control instructions. These operation task control instructions can include the position change, rotation angle, and gripper opening / closing state of the robot arm actuator, etc., to drive the robot to perform complex operation tasks such as grasping, sorting, and assembly.

[0091] The robot operation task control method based on visual language action model in this example embodiment will be described in further detail below.

[0092] In some example implementations, such as Figure 3 As shown, by combining the inter-viewpoint weights and the intra-viewpoint weights of each image region, the feature vectors of the retained regions in each visual image data are fused to obtain a fused feature vector. This includes: multiplying the inter-viewpoint weights of each visual image data with the corresponding intra-viewpoint weights of each image region to obtain the global importance score of each image region; identifying image regions with global importance scores greater than a preset score threshold as retained regions; weighting the local feature vectors of each retained region according to the global importance score to obtain the weighted feature vector of each retained region; and concatenating the weighted local feature vectors of each retained region in each visual image data to obtain the fused feature vector.

[0093] This process involves obtaining the inter-viewpoint weights for each visual image data point, output by the inter-viewpoint importance prediction network, and the intra-viewpoint weights for each local image region, output by the intra-viewpoint importance prediction network. For any given image region, the inter-viewpoint weights of its respective viewpoint are multiplied by its own intra-viewpoint weights to obtain its global importance score. This global importance score directly reflects the relative importance of the image region in a multi-viewpoint global context. This step achieves weight coupling across viewpoints and within local intra-viewpoint spaces, ensuring that both macroscopically important visual image data and microscopically more important image regions are preserved during feature processing, preventing key features from being averaged out or buried.

[0094] Image regions in each visual image whose global importance score is greater than a preset score threshold can be identified as retained regions, while image regions in each visual image whose global importance score is less than or equal to the preset score threshold can be removed. The preset score threshold can be determined based on actual circumstances, and this disclosure does not impose any special limitations.

[0095] The global importance score of each visual image data point can be used as a weighting coefficient, multiplied by the value of each feature dimension in the local feature vector of each retained region, to obtain the weighted feature vector of each retained region. During this calculation, for high-importance regions, such as a robotic arm grasping an object, the global importance score is high, and the weighted feature vector value remains unchanged or is relatively enhanced, thus preserving the features of more important image regions. For low-importance regions, such as irrelevant background clutter or occluded viewpoints, the global importance score is low, and the weighted feature vector value is reduced to near zero. By weighting the local feature vectors, the influence of high-importance image regions in the fusion feature can be increased, while the interference of background or low-weight image regions can be suppressed. This results in the weighted local feature vectors strengthening the feature values ​​of key regions, providing more accurate data support for action execution in complex tasks.

[0096] All weighted local vectors in the visual image data corresponding to the same viewpoint can be concatenated and stitched together according to their spatial order and viewpoint arrangement in the original visual image data to form a fused feature vector containing all image regions from multiple viewpoints. By fusing local feature vectors from multiple viewpoints and regions, a fused feature vector with strong consistency with the visual image data and high density of key information is obtained. This provides comprehensive and reliable perceptual input for the subsequent robot operation task control strategy network, improving the robot's operational accuracy in multi-source perception environments.

[0097] In some example implementations, such as Figure 3 As shown, by combining the inter-viewpoint weights and the intra-viewpoint weights of each image region, the feature vectors of the preserved regions in each visual image data are fused to obtain a fused feature vector, including the following steps S301 to S302: Step S301: Based on the inter-viewpoint weights and intra-viewpoint weights, pruning is performed on the image regions in each visual image data to obtain the retained regions in each visual image data.

[0098] It can obtain the inter-view weights output by the inter-view importance prediction network and the intra-view weights output by the intra-view importance prediction network. Based on the inter-view weights and intra-view weights, the retained regions can be determined, and other image regions besides the retained regions can be pruned.

[0099] Pruning can effectively remove unimportant or non-critical low-weight image data from each visual image dataset, reducing interference from subsequent invalid data and improving computational efficiency.

[0100] Step S302: The feature vectors of the preserved regions in the visual image data are fused to obtain the fused feature vector.

[0101] After pruning to obtain the retained regions, the feature vector of each retained region can be obtained. As an example, for multiple retained regions from the same viewpoint, the local feature vectors of each retained region can be concatenated into a long vector, and this long vector can be determined as the feature vector of the retained region; or the local feature vectors of each retained region can be weighted and averaged according to the importance score of each retained region to obtain the feature vector of the retained region.

[0102] Feature vectors of preserved regions from visual image data from different perspectives can be fused. For example, feature vectors of preserved regions in each visual image can be concatenated to obtain a fused feature vector.

[0103] Because the preserved regions have undergone pruning, the length of the feature sequence obtained by fusing the feature vectors of each preserved region will be significantly shorter than the length of the original full feature sequence. Moreover, it only contains visual semantic information with high importance scores, without containing redundant background information.

[0104] In some example implementations, image regions in each visual image data are pruned based on inter-viewpoint weights and intra-viewpoint weights to obtain retained regions in each visual image data. This includes: using the intra-viewpoint weights of the image regions in each visual image data to perform local pruning on each image region to obtain primary retained regions for each visual image data; calculating the global importance score of each image region in each primary retained region based on the inter-viewpoint weights and intra-viewpoint weights; and performing global pruning on each primary retained region in all visual image data based on the global importance score of each image region in each primary retained region to obtain retained regions in each visual image data.

[0105] like Figure 4 As shown, it can receive visual image data from different perspectives, as illustrated by the three sets of images at the top of the figure, representing the left, middle, and right perspectives, respectively. A visual encoder can be used to divide each image into image regions and map them to feature sequences of initial image block tokens. As shown in the figure, the three perspectives together generated a feature sequence of 768 initial image block tokens, containing a large amount of background redundancy information, such as blank desktops and distant walls.

[0106] Local pruning can be performed first. Within each viewpoint, background region tokens with prediction weights below a preset viewpoint threshold are identified and removed based on the prediction weights within that viewpoint. As shown by the scissors icon and the intermediate layer image, after local pruning, the original complete rectangular image is cropped into irregularly shaped primary retention regions. At this point, most irrelevant background has been removed, retaining only image block tokens containing the robot arm, target objects such as bottles, and their neighborhood. After this step, the total number of image block tokens is reduced from 768 to 483, achieving initial compression of spatial redundancy.

[0107] After local pruning, global pruning can be performed, aggregating tokens from all remaining views from the previous stage into a global candidate pool. The global importance score of each primary retained region is determined based on the inter-view weights of the visual image data corresponding to each primary retained region, and all primary retained regions are then sorted and filtered. As shown in the bottom image, this step further eliminates tokens containing objects but with low global value, such as severely occluded view remnants or edge features that contribute little to the current operation. The final number of retained image patch tokens is reduced to 241, resulting in the retained regions.

[0108] This embodiment employs a hierarchical pruning process, first local and then global, which not only preserves the features of key regions corresponding to each viewpoint but also achieves physical-level dimensionality reduction of visual data, reducing the number of computations in the subsequent policy network and improving the inference speed of the robot control system. By progressively removing background noise and invalid viewpoint interference, the policy network can focus on processing high-value task-related features, thereby improving the model's action prediction accuracy and robustness in complex environments.

[0109] In some example implementations, based on inter-viewpoint weights and intra-viewpoint weights, image regions in each visual image data are pruned to obtain retained regions in each visual image data. This includes: obtaining the inter-viewpoint weights corresponding to each visual image data, identifying visual image data with inter-viewpoint weights greater than a preset inter-viewpoint threshold as candidate viewpoint data; determining the retained regions in each candidate viewpoint data according to the sorting results, and filtering the image regions of the retained regions from each candidate viewpoint data. The inter-viewpoint prediction weights corresponding to each visual image data can be obtained and compared with a preset inter-viewpoint threshold. If the inter-viewpoint prediction weight corresponding to a certain visual image data is less than or equal to the preset inter-viewpoint threshold, the viewpoint is determined to be an invalid viewpoint, such as being completely occluded or facing away from the operation area, and all image regions contained in the visual image data are directly removed without further processing; if it is greater than the preset inter-viewpoint threshold, the visual image data is retained and identified as candidate viewpoint data. The preset inter-viewpoint threshold can be determined according to actual conditions, and this disclosure does not impose any special limitations.

[0110] For each retained candidate viewpoint data, the intra-viewpoint prediction weights corresponding to each image region within it are obtained. The image regions can be sorted from highest to lowest according to their intra-viewpoint prediction weights, and the top K image regions are selected as retained regions; alternatively, the intra-viewpoint prediction weights can be compared with a preset intra-viewpoint threshold, and image regions with weights higher than the preset threshold are determined as retained regions, while image regions with weights lower than the preset threshold are removed. The preset intra-viewpoint threshold can be determined based on actual conditions, and this disclosure does not impose any special limitations.

[0111] The local feature vectors corresponding to the remaining image regions after the above two-level filtering can be obtained, thus obtaining the set of local feature vectors of the retained regions in each visual image data.

[0112] By prioritizing the use of inter-viewpoint weights for coarse-grained screening, the entire visual image data that is deemed invalid or of low value can be directly eliminated, which can significantly reduce the initial scale of data processing. Computational resources can be concentrated on the candidate viewpoint data that has passed the screening, effectively reducing processing latency. Subsequent pruning based on intra-viewpoint weights can further remove redundant areas, ensuring the accuracy of key information in the final retained areas.

[0113] like Figure 5 As shown, based on local feature vectors, a pre-trained in-view importance prediction network is used to predict the in-view weights of the corresponding image regions, including the following steps S501 to S504: Step S501: Use the in-view importance prediction network to perform initial scoring on the local feature vectors to obtain the original importance score of the image region to be predicted.

[0114] The local feature vector corresponding to each image region can be independently input into a pre-trained intra-view importance prediction network. This pre-trained intra-view importance prediction network can identify semantic features contained in the local feature vectors, such as whether it belongs to a robot arm actuator, a target object, or the environmental background. The intra-view importance prediction network can output a corresponding scalar value for each local feature vector, and this scalar value is determined as the original importance score of the image region to be predicted.

[0115] By scoring each image region, information about key areas that are of high importance to the robot's operation tasks can be preliminarily identified.

[0116] Step S502: Calculate the spatial distance between the image region to be predicted and other image regions in the visual image data.

[0117] For any image region to be predicted, its corresponding visual image data has been divided into a regular grid array during the preprocessing stage. Each image region has fixed spatial index coordinates in the feature map corresponding to the visual image data. The spatial index coordinates of the image region to be predicted in the corresponding visual image data can be obtained, and all other image regions in the visual image data except for the predicted image region can be traversed. The spatial distance between its spatial index coordinates and the spatial index coordinates of other image regions is calculated. Here, "other image regions" refers to all image regions in the visual image data other than the image region to be predicted.

[0118] By incorporating spatial relationships within visual image data, since target entities such as grippers or workpieces in robot operations are typically distributed continuously in physical space, introducing spatial distance can effectively eliminate isolated noise points and ensure the continuity of the detected targets.

[0119] Step S503: The original importance scores are smoothed and weighted using spatial distance to obtain the processed importance scores.

[0120] Based on the aforementioned spatial distance, the original importance score can be weighted and smoothed. For example, the influence of other image regions on the image region to be predicted can be determined by the spatial distance between them. The original importance of the predicted image region can then be weighted according to this influence to obtain the processed importance score. If the original importance score of an image region to be predicted is high, and the original importance scores of its adjacent image regions are also high, then the original importance score of the image region to be predicted will be further improved after processing. Conversely, if the original importance scores of its adjacent image regions are low, then the original importance score of those adjacent image regions will be reduced after processing. This processing method allows the importance score of the image region to be predicted to incorporate information from other image regions, while avoiding the influence of single-point anomalies on the image region to be predicted.

[0121] Step S504: Normalize the importance scores of each image region in the visual image data to obtain the in-view weight of the image region to be predicted.

[0122] The importance scores of each image region in visual image data can be normalized, locking the score of each image region within a preset numerical range, such as 0 to 1, thus obtaining the in-view weight corresponding to each image region. The magnitude of this in-view weight can intuitively reflect the probability that the image region contains key entities, such as a robot arm or an object being manipulated, from the current viewpoint.

[0123] Normalization ensures the comparability of weights between different viewpoints and image regions, providing standardized data input for joint pruning or weighted fusion by combining the weights between viewpoints, and improving the training accuracy of multi-view visual language action models.

[0124] In some example implementations, the original importance score is smoothed and weighted using spatial distance to obtain a processed importance score. This includes: determining the ratio of the original importance score of other image regions to the corresponding spatial distance as the distance attenuation term for the other image regions; determining a correction value based on the weighted sum of the distance attenuation terms for the other image regions; and determining the processed importance score by summing the correction value with the original importance score.

[0125] As an example, the original importance scores of other image regions and the spatial distance between the image region to be predicted and other image regions can be obtained. The ratio between the original importance scores of other image regions and their corresponding spatial distances can be determined as the distance attenuation term for those other image regions. The closer the region is to the image region to be predicted, the larger the value of the distance attenuation term; conversely, the farther the region is from the image region to be predicted, the smaller the value of the distance attenuation term, even approaching zero. For example, the distance attenuation term can be calculated using the following formula: in, This represents the importance score of region n in the image to be predicted after processing. This represents the original importance score of other image regions i. This represents the spatial distance between image region n and image region i. To prevent the default constant from having a denominator of zero, the default constant can be a very small value, and this disclosure does not impose any special restrictions.

[0126] The correction value can be determined by the weighted sum of the distance attenuation terms corresponding to all other image regions. The weights corresponding to other image regions can be determined according to the actual situation, and this disclosure does not impose any limitations. As an example, the first value can be determined by the sum of the reciprocals of the spatial distances between other image regions and the image region to be predicted, and the reciprocal of the first value can be determined as the normalized weight. Each distance attenuation term is multiplied by this normalized weight, and the sum of the products after each multiplication is determined as the correction value. Furthermore, in the specific calculation process, a neighborhood radius can be set. If the spatial distance between other image regions and the image region to be predicted is greater than the neighborhood radius, then this weight does not need to be calculated and can be excluded from subsequent calculations. This neighborhood radius can be determined according to the actual situation, and this disclosure does not impose any special limitations. By setting this neighborhood radius, other image regions that are relatively close to the image regions to be predicted can be selected for local smoothing processing, thereby reducing the computational load and eliminating interference from long-distance noise. This correction value is used to indicate the degree of influence of other image regions on the image region to be predicted. If there are important key entities around the image region to be predicted, such as a robot arm, the correction value will be higher.

[0127] The original importance score and the correction value of the image region to be predicted can be summed to obtain the final processed importance score. For example, if the image region to be predicted has a low importance score due to reflection, but there are image regions with high importance scores in its surrounding neighborhood, such as inside a robot arm, a higher correction value can be provided to the image region to be predicted, thereby improving its importance score.

[0128] By assessing the influence of image regions outside the image region to be predicted on that image region, the final importance score of the image region to be predicted can be calibrated, transforming isolated importance score evaluations into spatially constrained evaluations and effectively improving the accuracy of importance score evaluations.

[0129] In some example implementations, based on each global feature vector, a pre-trained inter-view importance prediction network is used to predict the inter-view weights of the corresponding visual image data. This includes: concatenating the global feature vectors of each visual image data to construct a multi-view context vector; inputting the multi-view context vector into the inter-view importance prediction network, and outputting initial prediction values ​​through the inter-view importance prediction network; and normalizing the initial prediction values ​​to obtain the inter-view weights of each visual image data.

[0130] After obtaining the global feature vectors output by the pre-trained visual encoder, each corresponding to a different viewpoint of the robot, these discrete feature vectors can be integrated into a unified semantic space. For example, they can be arranged according to a preset viewpoint order, such as the main viewpoint, left wrist viewpoint, right wrist viewpoint, etc., and the global feature vectors of each visual image data can be concatenated along the feature dimension. Assuming each global feature vector has a dimension of D, and the robot has N visual viewpoints, through the above concatenation operation, the system connects the N independent global feature vectors end-to-end to construct a long vector of dimension N*D. This long vector is the multi-view context vector. Structurally, this multi-view context vector simultaneously preserves the global semantic information of all viewpoints and their relative positional relationships.

[0131] The constructed multi-view context vectors can be used as input data and directly fed into a pre-trained inter-view importance prediction network. This pre-trained network performs non-linear mapping and feature calculation on the input multi-view context vectors and outputs a set of initial prediction values. The number of these initial prediction values ​​is equal to the number of views N, and each value corresponds to one view. For example, the initial prediction values ​​can be calculated using the following formula: in, This represents the initial predicted value of the output. This represents a network for predicting the importance of views. This represents the multi-view context vector obtained by concatenating the global feature vectors of all viewpoints. This set of initial predicted values ​​can be normalized, mapping them to the probability distribution interval (0,1). Each value obtained after normalization is then determined as the viewpoint weight of the corresponding visual image data.

[0132] In some example implementations, after the initial predicted values ​​are output by the inter-view importance prediction network, before normalizing the initial predicted data, the spatial relative distance between the robot's current end effector and the task entity to be operated can be obtained; an adaptive distance weight factor for each visual image data is determined based on the spatial relative distance; the inter-view weights output by the inter-view importance prediction network are corrected using the adaptive distance weight factor to obtain corrected predicted data; and the corrected predicted data is normalized based on the Softmax function, mapping it to the probability distribution interval (0, 1) to obtain the final inter-view weights.

[0133] As an example, the real-time 3D coordinates of the robot arm's end effector can be obtained; at the same time, the target of the task entity in the current view can be located, and the 3D coordinates of the center point or key point of the task entity can be determined; the relative distance between the 3D coordinates of the robot arm's end effector and the 3D coordinates of the task entity can be determined.

[0134] When the relative distance is greater than a preset interaction distance threshold, the adaptive distance weighting factors for each viewpoint can be set to balanced initial values. As the relative distance gradually decreases and becomes less than or equal to the preset interaction distance threshold, the robot may enter a refined operation phase, at which point the adaptive distance weighting factors for each viewpoint can be calculated according to preset dynamic mapping rules. For example, for the viewpoint of the wrist camera mounted at the end of the robot arm, its corresponding adaptive distance weighting factor can increase non-linearly as the relative distance decreases; while for the viewpoint of the global camera far from the interaction area, its corresponding adaptive distance weighting factor decreases accordingly. Through this dynamic mapping rule, the adaptive distance weighting factor corresponding to each visual image data at the current operation moment can be determined.

[0135] The corrected inter-view weights can be obtained by weighting the adaptive distance weight factors corresponding to each viewpoint with the inter-view importance prediction network output. For example... Figure 10 As shown, in Figure 10 In the diagram, the horizontal axis represents the frame rate, indicating the time progress of the task; the vertical axis represents the weight between different viewpoints, with a higher value indicating a greater influence of that viewpoint on the robot's decision-making at the current moment. Specifically, the main view camera is typically mounted on or above the robot's head; the left-side view camera is typically mounted on the left arm or left side; and the right-side view camera is typically mounted on the right arm or right side.

[0136] like Figure 10As shown, the curve corresponding to the main view camera remains consistently at 1.0. This indicates that the main view, as a global reference, provides essential environmental background and global spatial information throughout the entire operation. The curve corresponding to the left view camera rapidly climbs and remains high in the early stages of the task (approximately frames 50-110). At this time, the left robotic arm is approaching the task entity, allowing for an increase in the weight of the left view to capture precise interaction details. The curve corresponding to the right view camera surges in the later stages of the task (approximately frames 190-270) as the right robotic arm intervenes, adaptively shifting attention from the left to the right view to assist in completing the finer movements on the right side.

[0137] This correction method causes the image weights containing the robot arm's perspective to generate a distinct peak when entering the interaction phase, thereby achieving precise focusing on key perspectives.

[0138] Overall, this embodiment improves the decision-making accuracy in the fine-grained operation stage by introducing physical distance to correct the weights. Especially at critical moments of object interaction, it ensures that the model prioritizes extracting the perspective features that best reflect the hand-object relationship. At the same time, it avoids the oscillation phenomenon of frequent weight jumps during multi-view fusion, ensures the continuity of robot action command output, and improves the overall accuracy and efficiency of operation task execution.

[0139] In some example implementations, feature extraction is performed on each image region in each visual image data to obtain multiple local feature vectors corresponding to the visual image data. This includes: inputting the sequence of each image region in each visual image data into a pre-trained visual encoder, processing each image region using the self-attention mechanism of the pre-trained visual encoder, generating a feature sequence including a category token and an image block token; separating the image block token from the feature sequence, and determining each local feature vector corresponding to the visual image data based on the image block token corresponding to each image region.

[0140] The pre-trained visual encoder can employ a Transformer-based visual model, such as the Vision Transformer (ViT) architecture or the Sigmoid Language-Image Pre-training (SigLIP) architecture. During the input phase, the visual image data from multiple robot perspectives can be divided into multiple fixed-size image regions. Each image region is mapped to a one-dimensional vector, which serves as the image patch token, representing the local detail features of each image region. An initial category token can be inserted at the beginning of this sequence of image patch tokens. This initial category token, along with all the image patch tokens, forms an initial token sequence. This initial token sequence can be processed using the attention mechanism in the pre-trained visual encoder. For example, attention can be used to calculate the association between the initial category token and each image patch token, and the feature values ​​of the initial category token and each image patch token can be updated. The pre-trained visual encoder then outputs the processed feature sequence.

[0141] The feature sequence output by the pre-trained visual encoder is a set of multiple feature vectors. Because the pre-trained visual encoder employs a self-attention mechanism, the category token at a specific position in the feature sequence, such as the first position, already aggregates the global semantic information of the visual image data. The category token can be separated from the feature sequence and identified as the global feature vector corresponding to the visual image data. For each viewpoint of the robot, the above generation and separation steps can be performed using the pre-trained visual encoder to obtain the global feature vector corresponding to each viewpoint. By utilizing the pre-trained visual encoder, the first category token can directly capture long-range semantic dependencies in the visual image data. The first category token generated by the pre-trained visual encoder can be directly used as a global feature vector representing the information of the entire image without increasing additional computational overhead.

[0142] By using predefined indexing rules, category tokens representing global information can be removed from the feature sequence, separating the remaining vectors. These separated vectors are the deeply encoded image patch tokens. Since they have a one-to-one spatial mapping relationship with the grid division positions of the original image, these image patch tokens can be identified as multiple local feature vectors corresponding to the visual image data. These local feature vectors together constitute a feature map rich in semantic information, accurately reflecting the semantic information of each local image region.

[0143] The training method of the robot operation task control model based on the visual language action model in this example embodiment will be described in detail below. This model includes at least an inter-view importance prediction network, an intra-view importance prediction network, and a policy network. (Reference) Figure 2 As shown, the method may include the following steps S601 to S604: Step S601: Determine the ground truth labels for importance between viewpoints and importance within viewpoints based on the training data.

[0144] A sample dataset for model training can be obtained. This training data includes visual image data from multiple perspectives during robot task execution, robot action labels corresponding to each visual image data, and robot state description information. The visual image data can include global and local view image data. Global view image data can be acquired by a fixed camera mounted externally to the robot, representing the overall layout during action execution. Local view image data can be acquired by a wrist camera mounted on the robot arm actuator, representing specific operational details during action execution. The robot action labels include the pose changes of the robot arm actuator at the corresponding moment for each visual image data, such as position offset and posture rotation; and gripper control commands, such as the opening and closing state of the gripper or the target width value. The robot state description information represents the three-dimensional coordinates of the target object, the robot's joint angles, and the three-dimensional coordinates of the robot arm actuator's end effector.

[0145] After acquiring the training data, ground truth labels for the importance between viewpoints can be determined. If the ground truth label for the importance between viewpoints is the first value, such as 1, it means that the 3D coordinates of the target object are projected into the effective range of the 2D image plane in the robot's viewpoint and are not occluded. If the ground truth label for the importance between viewpoints is the second value, such as 0, it means that the 3D coordinates of the target object are projected into the effective range of the 2D image plane in the robot's viewpoint or are occluded.

[0146] The importance truth value label within a viewpoint can be determined. If the importance truth value label within the viewpoint is the first value, such as 1, it indicates a critical region related to the execution of the task. If the importance truth value label within the viewpoint is the second value, such as 0, it indicates a region unrelated to the execution of the task.

[0147] Step S602: Use the inter-view importance prediction network to determine the inter-view prediction weights based on the global feature vector of the visual image data, and use the intra-view importance prediction network to determine the intra-view prediction weights based on the local feature vector of the visual image data.

[0148] A pre-trained visual encoder can extract global feature vectors (class tokens) and local feature vectors (image patch tokens) from each visual image data. These global feature vectors from each viewpoint can be input into an inter-view importance prediction network. This network processes the global feature vectors to obtain a score value for each viewpoint and normalizes these scores to obtain inter-view prediction weights, which represent the probability of each viewpoint contributing to the current task. Simultaneously, local feature vectors from each viewpoint can be input into an intra-view importance prediction network. This network processes the local feature vectors corresponding to each image region to obtain score values ​​and normalizes them to obtain intra-view prediction weights, which represent the probability that each local region in the image belongs to a key region.

[0149] The inter-view importance prediction network can be constructed as a multilayer perceptron structure. It can contain an input layer, at least one hidden layer, and an output layer. The input layer has the same dimension as the global feature vector. The hidden layers are configured with non-linear activation functions, such as ReLU. The output layer has a dimension of 1, or the same as the number of views, and is used to output the inter-view prediction weights of the global feature vector for that view.

[0150] Intra-view importance prediction networks can be constructed as point-to-point convolutional networks or multilayer perceptrons with shared weights. They can process the local feature vectors corresponding to each image region in parallel. This intra-view importance prediction network shares the same set of weight parameters across all image regions, mapping the feature vector of each image region to the probability value that the region belongs to a key region. The output layer uses a sigmoid activation function to ensure that the intra-view prediction weights are within the range of 0 to 1.

[0151] Step S603: Combine the inter-view prediction weights and intra-view prediction weights to perform weighted fusion of local feature vectors, and use the policy network to obtain the predicted action based on the fused feature vector.

[0152] It can obtain the local feature vectors and in-view prediction weights corresponding to each visual image data. The in-view prediction weights are then multiplied element-wise with their corresponding local feature vectors to obtain weighted local feature vectors. Through this operation, features of image regions with higher in-view prediction weights, such as target objects, are preserved or enhanced, while features of image regions with lower in-view prediction weights, such as desktop textures and walls, are weakened or zeroed out. For example, the local feature vectors can be calculated using the following formula: in, A local feature vector used to represent the nth image region from the vth viewpoint. This represents the inter-view weights from that perspective. This indicates the weight within the viewpoint of the region in the image after smoothing.

[0153] For each viewpoint, the inter-viewpoint prediction weights are used to weight the local feature vectors that have been weighted within the viewpoint, resulting in a weighted feature vector. The weighted feature vectors corresponding to each viewpoint can be concatenated to construct a multi-viewpoint context feature sequence that includes all viewpoints.

[0154] The preprocessed text embedding vector, such as the text embedding vector corresponding to the user's input text command, such as "grab the red square", can also be concatenated with the above multi-view context feature sequence to generate a fused feature vector.

[0155] The robot operation task control policy network can be built based on the decoder of the Transformer architecture. Each decoder layer can contain a multi-head attention sub-layer, a multi-head cross-attention sub-layer, and a feedforward neural network sub-layer. In this system, the fused feature vector serves as the Key and Value of the cross-attention sublayer, while the predicted action or preset query vector from the previous time step serves as the Query. The output head includes a fully connected layer for predicting the position and a binary classification layer for predicting the gripper state. When processing the input fused feature vector, the robot operation task control strategy network obtains a cross-attention weight map based on the dot product similarity between the corresponding query vector and the key vector. Based on the cross-attention weight map, the value vector is weighted and aggregated to obtain the current action query intent. This intent is then input into the feedforward neural network sublayer via the feature vector output from the cross-attention sublayer for nonlinear transformation, resulting in a decoded feature vector. This decoded feature vector is input into the output head, where the fully connected layer performs regression calculations to obtain the pose change at the end effector of the robot arm. The binary classification layer in the output head then performs classification calculations on the decoded feature vector to obtain the probability of the robot gripper's open / closed state. Combining the pose change and the open / closed state probabilities, the predicted action for the current time step is determined. The pose change can include the positional offset and rotation of the robot arm actuator's end effector in three-dimensional space; the gripper control command can include the probability of the gripper's open / closed state or the target width value. By employing a Transformer-based decoder to construct the policy network and utilizing a multi-head cross-attention mechanism to process the fused feature vector, the system can actively retrieve and focus on local visual features strongly correlated with the current action stage from the high-dimensional fused feature vector, rather than passively receiving all visual information. This effectively filters out irrelevant background interference in complex environments, ensuring that the robot's operation task control policy network accurately generates predicted actions, thereby improving the accuracy of robot operation task action execution.

[0156] In step S604, the parameters of the inter-view importance prediction network, the intra-view importance prediction network, and the policy network are updated according to the first auxiliary loss between the inter-view prediction weights and the inter-view importance ground truth labels, the second auxiliary loss between the intra-view prediction weights and the intra-view importance ground truth labels, and the action prediction loss between the predicted action and the robot action label.

[0157] The interview prediction weights and ground truth labels of interview importance output by the interview importance prediction network can be obtained. The difference between the interview prediction weights and the ground truth labels of interview importance can be calculated, and this difference can be determined as the first auxiliary loss.

[0158] The in-view prediction weights and ground truth labels of in-view importance output by the in-view importance prediction network can be obtained. The difference between the in-view prediction weights and the ground truth labels of in-view importance can be calculated element by element, and the mean of the difference of each image patch can be determined as the second auxiliary loss.

[0159] For pose changes in robot movements, the regression error between predicted and true values ​​can be calculated; for gripper opening and closing states in robot movements, the classification error between predicted probabilities and true labels can be calculated. The weighted sum of regression and classification errors can be determined as the motion prediction loss.

[0160] The first auxiliary loss, second auxiliary loss, and action prediction loss can be weighted and summed according to preset hyperparameter weights to obtain the total loss value. Based on the total loss value, the network gradient is calculated using the backpropagation algorithm, and an optimizer, such as the AdamW optimizer, is used to simultaneously update the parameters of the inter-view importance prediction network, the intra-view importance prediction network, the policy network, and the front-end visual encoder until the total loss value converges to a preset threshold range. This preset threshold range can be limited according to actual conditions, and this disclosure does not impose any special limitations. For example, the total loss value can be calculated using the following formula: in, This represents the total loss value. Indicates the loss from action prediction. Indicates the first auxiliary loss. This indicates the second auxiliary loss. and This represents the preset hyperparameter weighting coefficients.

[0161] Traditional end-to-end training relies solely on action prediction loss, which can easily lead to the model getting stuck in local optima. This embodiment uses a first auxiliary loss to force the model to learn to distinguish between effective and occluded viewpoints, and a second auxiliary loss to force the model to focus on key areas such as the end effector of the robot arm and the target object. This accelerates the model's convergence speed and improves its robustness and generalization ability in the face of complex backgrounds or dynamic occlusion during the testing phase.

[0162] In some example implementations, determining ground truth labels for inter-view importance and intra-view importance based on training data includes: determining ground truth labels for inter-view importance of each visual image data based on robot state description information corresponding to each visual image data; and generating ground truth labels for intra-view importance of each visual image data based on the correspondence between each image region and the task entity in each visual image data.

[0163] The current joint angle data of the robot can be extracted from the robot state description information, and the three-dimensional coordinates of the end effector of the robot arm can be determined using the joint angle data; or, the three-dimensional coordinates of the target object in the robot state description information can be obtained.

[0164] For each viewpoint, obtain the extrinsic and intrinsic parameter matrices of the camera corresponding to that viewpoint. Use the extrinsic parameter matrix to transform the 3D coordinates of the robot arm's end effector or the target object to the camera's coordinate system, and use the intrinsic parameter matrix to project it to the pixel coordinate system to obtain the 2D projection point coordinates.

[0165] It can be determined whether the coordinates of the two-dimensional projection point are within the image resolution range of the viewpoint. If the coordinates of the two-dimensional projection point are within the image resolution range of the viewpoint, and the coordinates of the two-dimensional projection point are not occluded according to the depth information, then the viewpoint is determined to be a valid viewpoint, and the corresponding viewpoint importance truth value label is set to the first value, such as 1. If the projection point is outside the image resolution range of the viewpoint, or is determined to be invisible, then the viewpoint is determined to be an invalid viewpoint, and the corresponding viewpoint importance truth value label is set to the second value, such as 0.

[0166] It can obtain the geometric bounding box of the task entity in three-dimensional space, project the coordinates of all its vertices onto the two-dimensional image plane of each viewpoint, and calculate the smallest convex polygon region or mask region containing these projected vertices. This region is the projection region of the task entity in the two-dimensional image plane.

[0167] Visual image data can be divided into multiple regularly arranged image regions. For each image region, the overlap ratio between its pixel range and its corresponding projection region is calculated. This overlap ratio can be compared with a preset ratio threshold: if an image region overlaps with its corresponding projection region and the overlap ratio is greater than the preset ratio threshold, it indicates that the image region contains the visual features of the task entity. The ground truth label of the viewpoint importance of this image region is determined as the first value, such as 1, indicating that the image region is a key region related to the task and has high attention. If an image region does not overlap with its corresponding projection region or the overlap ratio is less than or equal to the preset ratio threshold, it indicates that the image region is a background region. The ground truth label of its viewpoint importance is determined as the second value, such as 0, indicating that the image region is a non-key region unrelated to the task and has low attention.

[0168] By generating ground truth labels using robot state description information and the geometric positional relationship of task entities, on the one hand, the generation of inter-view labels can eliminate invalid data frames caused by dynamic occlusion due to robot arm movement or targets moving out of the field of view, preventing the model from learning incorrect global features; on the other hand, the generation of intra-view labels establishes the correspondence between image regions and task entities, enabling the model to maintain feature extraction accuracy in complex environments even without manual pixel-level annotation.

[0169] In some example implementations, the importance truth labels between viewpoints of each visual image data are determined based on the robot state description information corresponding to each visual image data. This includes: determining the current operation stage of each robot based on the robot state description information; if the operation stage is an object interaction stage, then the importance truth labels between viewpoints of the visual image data corresponding to the main viewpoint and the visual image data containing at least one robot arm viewpoint are marked as first values; if the operation stage is a non-interaction stage, then the importance truth labels between viewpoints of the visual image data corresponding to the main viewpoint are marked as first values, and the importance truth labels of the visual image data containing the robot arm viewpoint are marked as second values.

[0170] like Figure 7The left side shows a large language model, such as GPT (Generative Pre-trained Transformer System), which converts trajectory data, pose data, gripper state, and 3D coordinates of the target object from the robot arm's end effector into text-based state description information. Based on this state description, the robot's current state is determined, such as determining that the left gripper is stationary and the right gripper is working. A ground truth label for the importance between viewpoints is also determined based on this state. For example, the 3D coordinates of the robot arm's end effector and the target object can be extracted from the robot's state description information. The Euclidean distance between these two coordinates is calculated and compared to a preset distance threshold. If the Euclidean distance is less than or equal to the preset threshold, or the robot gripper is closed (indicating object grasping), the robot is considered to be in an object interaction phase, such as grasping, placing, or assembling. If the Euclidean distance is greater than the preset threshold and the robot gripper is open, the robot is considered to be in a non-interaction phase, such as approaching, moving, or returning to its original position. In addition to determining the state based on the robot's state description information, other methods can also be used... Figure 7 The other two methods shown on the left are manual annotation and bounding box detectors, such as acquiring the 2D bounding boxes of the robot arm and the manipulated object in real time. The overlap rate of the two bounding boxes is calculated. If overlap exists, it is determined that an interaction is taking place.

[0171] When the robot is in the object interaction phase, the first-person perspective provides information about the environmental background and obstacles, while the robot arm's perspective provides information about the relative position between the robot arm's end effector and the object. Therefore, both perspectives can be considered valid. The ground truth labels for the inter-perspective importance of the visual image data corresponding to the first-person perspective can be set to the first value, and the ground truth labels for the inter-perspective importance of the visual image data corresponding to the robot arm's perspective can also be set to the first value. At this point, the model is forced to simultaneously focus on both the global environment and local details.

[0172] When the robot is in a non-interactive phase, such as rapidly moving towards a target, the camera mounted on the robot arm may capture motion blur or lack any valuable objects within its field of view, such as only the ground. Forcing focus on this perspective might introduce noise interference. Therefore, the ground truth label for the importance of different perspectives corresponding to the main viewpoint can be kept at the first value to maintain awareness of the global environment; simultaneously, the ground truth label for the importance of different perspectives corresponding to the robot arm's viewpoint can be modified to the second value. In this case, the model is guided to ignore the wrist camera's view during the movement phase and focus only on the global perspective. Figure 7As shown, the ground truth labels for the importance of the three perspectives are 1, 0, and 1, respectively. This achieves automatic masking of invalid perspectives, such as a stationary camera, accurately identifies key moments of hand-object interaction and preserves their features, achieves pixel-level removal of the background and floor, and retains only the task entity.

[0173] By dynamically dividing the interaction phase into interactive and non-interactive phases based on the distance relationship between the robot and the object, and accordingly setting ground value labels for the importance of different viewpoints, this approach aligns with human operational habits. In the non-interactive phase, reducing the weight label of the robot arm's viewpoint effectively shields the model from visual blurring or invalid fields of view caused by the robot arm's rapid movements, allowing the model to focus on global path planning. In the interactive phase, increasing the weight label of the robot arm's viewpoint forces the model to incorporate high-resolution details from close range. This enhances the model's robustness in long-sequence tasks and provides clear and physically consistent training guidance.

[0174] In some exemplary embodiments, generating a ground truth label for the viewpoint importance of each visual image data based on the correspondence between each image region and the task entity in each visual image data includes: obtaining the target region range of the task entity in each visual image data; determining the pixels in each visual image data that are within the corresponding target region range as first pixels, and marking the viewpoint importance of the first pixels as a first value; determining the pixels in each visual image data that are not within the corresponding target region range as second pixels, and marking the viewpoint importance of the second pixels as a second value.

[0175] like Figure 7 As shown on the right, a target segmentation model combined with text prompts can be used to detect targets in visual image data. By acquiring the input task entity, such as a robot arm or a manipulated object, the 3D bounding box vertex coordinates of the task entity can be determined. Using the camera's intrinsic and extrinsic parameter matrices, the vertex coordinates are projected onto the 2D image pixel coordinate system. Based on the projected vertex coordinates, a minimum convex polygon region or a binarized mask region is generated. This region is the target region. The pixels within this region represent the visual feature positions of the task entity.

[0176] Visual image data can be divided into multiple image regions. For each image region, the number or proportion of pixels within that region that belong to the target region is counted, i.e., the first pixel. If the number or proportion of the first pixel in an image region is greater than a preset overlap threshold, the image region is determined to be a key feature region, and its corresponding ground truth label of importance within the viewpoint is marked as the first value. The preset overlap threshold can be determined according to the circumstances, and this disclosure does not impose any special limitations.

[0177] If all pixels within a certain image region are not within the target region (i.e., all are second pixels), or the proportion of first pixels is less than or equal to a preset overlap threshold, then the image region is determined to be a background region. The ground truth label for the viewpoint importance of this background region can be marked as a second value.

[0178] Through the above steps, a binary label sequence with the same dimension as the number of image regions can be generated as a supervision signal for training the in-view importance prediction network. This can ignore background noise inside the image region and improve the accuracy of local feature extraction.

[0179] In some exemplary embodiments, the task entity includes at least a robotic arm actuator and the object being manipulated.

[0180] In this context, a robot arm actuator refers to the component at the end of a robot arm used to perform physical interactions, such as a two-finger gripper, a suction cup, or a dexterous hand. A three-dimensional bounding box surrounding the actuator can be generated based on its known geometric dimensions (CAD model). The projection area of ​​this bounding box onto the image represents the robot's visual location.

[0181] The manipulated object refers to the interactive target specified in the task instruction, such as a workpiece to be grasped, a nut to be tightened, or a container to be placed. By acquiring the 3D center coordinates and orientation of the manipulated object in real time, a corresponding 3D bounding box can be generated based on the object's physical dimensions. The projection area of ​​this bounding box on the image represents the target's visual location.

[0182] By forcing the model to focus its attention weights on the image regions of the two entities, the model can automatically ignore interference from irrelevant background objects in the scene and concentrate on analyzing the relative distance and relative posture between the robot arm and the manipulated object. This improves the motion prediction accuracy of the policy network during the contact phase and effectively avoids grasping failures or collisions caused by background noise.

[0183] In some exemplary embodiments, such as Figure 8 The diagram shows the operational architecture of a robot operation task control model based on a deep reinforcement learning visual language action model, as disclosed in this disclosure.

[0184] The robot operation task control model based on a deep reinforcement learning visual language action model is a robot control system architecture that integrates offline supervision and online dynamic pruning. It includes an offline two-layer importance labeling module, an online hierarchical pruning module (BFA++Prune), and a robot operation task control policy network. The two-layer importance labeling module can be used to generate ground truth labels for inter-view importance and intra-view importance offline. The hierarchical pruning module can perform hierarchical pruning on the input multi-view visual image data based on the global and local semantic vectors generated by the visual encoder to obtain high-value preserved regions and fused feature vectors of the preserved regions. The robot operation task control policy network is used to obtain operation task control commands for controlling the robot to perform operation tasks based on the fused feature vectors.

[0185] This involves acquiring multi-view visual image data of the robot and using a visual encoder to extract features from each visual image. For example... Figure 8 As shown, the feature sequence output by the visual encoder includes category tokens and image patch tokens. Category tokens represent the global feature vector for that viewpoint; image patch tokens represent the local feature vectors of each image region after the visual image has been divided into multiple image regions.

[0186] The interview importance prediction network can receive category tokens from each viewpoint, i.e., global feature vectors, as input, and predict the corresponding interview prediction weights; these interview prediction weights characterize the credibility of the current viewpoint in the overall task.

[0187] The intraview importance prediction network can receive image patch tokens (i.e., local feature vectors) from each viewpoint as input and predict the corresponding intraview prediction weights. These intraview prediction weights characterize the local saliency of each image region within each viewpoint, such as the robot arm, objects, and background.

[0188] During the training of the inter-view importance prediction network and the intra-view importance prediction network (dashed line process), ground truth labels for inter-view importance and intra-view importance can be generated based on the two-layer importance labeling module. The inter-view importance prediction network is updated by backpropagation based on the loss value between the inter-view prediction weights and the inter-view importance ground truth labels; the intra-view importance prediction network is updated by backpropagation based on the loss value between the intra-view prediction weights and the intra-view importance ground truth labels.

[0189] The inter-view importance prediction network can be used to determine the inter-view weights of each visual image data and the intra-view weights of each image region. A hierarchical pruning module can then dynamically filter the regions to be retained. For example... Figure 8As shown, local pruning can be used to remove the image region corresponding to the background by utilizing the intra-view weight of the image region within each viewpoint (as indicated by the scissors icon in the figure); then, global pruning can be used to calculate the global importance score by combining the inter-view weights, and image regions with low global importance scores can be removed across viewpoints.

[0190] The high-value image regions retained after double screening, along with the embedding vectors of natural language instructions, such as "put the banana in the blue plate," are input into the robot's task control policy network. This network infers based on the fused feature vectors and outputs task control instructions for the robot arm, such as grasping and placing motion commands.

[0191] As an example, the inventors verified the success rates of different methods in the execution of robot operation tasks during testing and verification, as shown in Table 1. In the table, hammering, hitting a wooden block, transferring a bottle, picking up a wooden block, stacking cups, hanging a bottle, adjusting shoes, and placing a cup represent different operation tasks. The corresponding values ​​for each operation task, such as 0.78, 0.52, 0.65, 0.54, 0.07, 0.46, 0.59, 0.36, and 0.495, represent the average success rate of each operation task under a specific model. The average value represents the average of all average success rates corresponding to all operation tasks under a specific model. The number of frames transmitted per second represents the inference speed of the model; the larger the value, the faster the model runs.

[0192] Table 1 In Table 1, The original model without pruning is represented by DART (Diffusion-based Adaptive), which represents a comparative data augmentation / processing method. BFA (Best-Feature-Aware Fusion for Multi-View Fine-grained Manipulation) is a predecessor technology of this disclosure. BFA++ (Best-Feature-Aware Fusion for Multi-View Fine-grained Manipulation) is a hierarchical pruning method based on two-layer importance labeling in this disclosure.

[0193] As can be seen from Table 1, compared with the unpruned or base model, the method disclosed herein improves the success rate by about 10%; the number of frames transmitted per second is increased by 1.5 to 1.9 times; and the number of floating-point operations is significantly reduced due to the substantial reduction in input tokens.

[0194] As shown in Table 2, the inventors compared the performance of different models in performing specific robot tasks in a real-world environment during testing and verification. In Table 2, bottle transfer, grabbing a toy chicken, sliding a bag, banana (easy), and banana (difficult) represent different tasks. The values ​​corresponding to each task, such as 0.3, 0.45, 0.15, 0.45, and 0.35, represent the average success rate of each task under a specific model. The closer the value is to 1.0, the higher the reliability of the model in completing the task. The overall success rate represents the average of all average success rates of the specific model under all tasks. The number of frames transmitted per second represents the inference speed of the model; the larger the value, the faster the model runs.

[0195] Table 2 According to Table 2, it can be determined that in the banana-difficult task, in Under this architecture, the provisions of this disclosure The method improves the average success rate from 0.35 to 0.55, demonstrating that the hierarchical pruning strategy of this disclosure can enhance the model's accuracy in recognizing and manipulating target entities by removing background noise in complex and multi-interference environments. In the sliding bag task, the proposed method... The method doubled the average success rate from 0.15 to 0.30, demonstrating that the dual-layer determination mechanism of inter-view importance and intra-view importance can effectively capture subtle physical interaction features between objects and the robot arm; In the model, the present disclosure proposes The method achieved a peak average success rate of 0.55 while maintaining a high-frequency control of 9.4 Hz. Although the frequency was slightly lower than that of BFA, which only performs in-view pruning, the introduction of a two-layer filtering logic in BFA++ balanced some efficiency, resulting in an improvement in the average success rate from 0.46 to 0.55. Under the proposed architecture, the average success rate of BFA++ in all operation tasks is 0.55, significantly higher than the 0.34 of the original model; under the RDT architecture, the average success rate of BFA++ in all operation tasks reaches 0.53, far exceeding the 0.29 of the original model. This demonstrates that the hierarchical pruning strategy proposed in this disclosure can improve operation accuracy.

[0196] During testing and verification, the inventor, such as Figure 9As shown, Figure (a) represents the original baseline model. The feature distribution of the model is shown in Figure (b), which illustrates the feature distribution of the method disclosed in this publication, namely BFA++, after hierarchical pruning. Here, the main view token, left view token, and right view token represent image patch tokens of image regions from different camera perspectives, i.e., local feature vectors of the image regions, while the language token represents the semantic feature vector of the task instruction.

[0197] from Figure 9 It can be determined that in the baseline model (a), the distribution of local feature vectors from different perspectives is relatively chaotic and has a large amount of overlap, indicating that there is serious redundant information interference between perspectives. In contrast, in the BFA++ model (b) of this disclosure, the local feature vectors of each perspective are highly aggregated and separated from each other. This proves that this disclosure effectively eliminates spatial redundancy through double-layer importance labeling, making the features extracted by the model have extremely high discriminative power. Comparing the two figures (a) and (b), it can be seen that the discriminative power of the feature space distribution after pruning is more obvious. This indicates that the hierarchical pruning strategy of this disclosure almost completely eliminates invalid background noise, such as redundant information like floors and walls, and only retains features related to the core of the task. This highly discriminative feature distribution allows the model to more keenly capture the complementary information provided by different perspectives without being misled by repetitive redundant data. In the figure (b), the semantic feature vector can more accurately form a logical mapping with the local feature vectors of each perspective. This consistency in spatial distribution explains why the method of this disclosure can quickly extract useful information from highly discriminative features when performing complex tasks such as banana-hard tasks, thereby significantly improving the average success rate from the original 0.35 to 0.55. The feature distribution changes from the chaotic interweaving in (a) to the categorization in (b), which physically and logically proves that this disclosure reduces the fitting difficulty of the policy backbone network by eliminating redundant information, enabling the model to better focus on the core features of hand-object interaction, and achieving a dual gain in performance and accuracy.

[0198] In testing and verification, the inventors conducted a comparative experiment to verify the information content carried by image block tokens, i.e., local feature vectors. The experimental results are shown in Table 3. This table demonstrates from an information theory perspective that the BFA++ hierarchical pruning method disclosed in this paper not only reduces the amount of data but also improves the uniqueness and information richness of the retained data.

[0199] Table 3 Average similarity measures the degree of repetition between different image patch tokens. A lower value indicates greater difference and less redundancy between image patch tokens. Original model The corresponding average similarity was as high as 0.928, indicating the presence of a large number of redundant image patch tokens with overlapping content; while BFA++ reduced this to 0.865. This proves that BFA++ effectively eliminates repetitive information between and within viewpoints, while preserving unique features with high discriminative power.

[0200] Average entropy is used as a metric to measure the richness of information. A higher value indicates a greater amount of effective information carried by a single image patch token. Table 3 shows that the entropy value of BFA++ reaches 0.791, significantly higher than the 0.637 of the original model and 0.731 of the randomly pruned model. This indicates that each image patch token selected by BFA++ in this disclosure contains more key operational features, rather than meaningless background noise.

[0201] The method disclosed herein increases the average entropy of image patch tokens from 0.637 to 0.791. Because the pruning process eliminates a large number of low-entropy background regions, each image patch token input to the policy network has a higher value density. Comparing the results with random pruning, it is clear that while simple random sampling can reduce similarity to some extent, it cannot achieve optimal information quality improvement through a two-layer importance determination like the BFA++ method of this disclosure.

[0202] In testing and verification, the inventors compared the impact of each core component in the BFA++ architecture provided in this disclosure on the success rate of robot operation tasks, as shown in Table 4. By removing specific components or changing processing methods, the completeness and necessity of the technical solution disclosed in this disclosure were verified. In Table 4, striking a wooden block, hanging a bottle, adjusting shoes, and placing a cup represent different operation tasks. The values ​​corresponding to each operation task, such as 0.76, 0.41, 0.56, and 0.50, represent the average success rate of each operation task after the absence or replacement of a specific component. The average success rate represents the average of all average success rates corresponding to all operation tasks after the specific component is removed.

[0203] Table 4 As can be seen from Table 4, compared to the average success rate of 0.565 after pruning without stratification and the average success rate of 0.535 after random drop, the average success rate of this disclosure reaches the highest of 0.638.

[0204] When the inter-viewpoint predictor (i.e., the inter-viewpoint importance prediction network) or the intra-viewpoint predictor (i.e., the intra-viewpoint importance prediction network) is missing, the average success rate of each operation task will decrease. This indicates that only by simultaneously starting from two dimensions, namely selecting the correct camera viewpoint and selecting the correct image region, can redundant interference be eliminated to the maximum extent and the model's performance in complex tasks such as hanging bottles be improved.

[0205] Experimental results show that by introducing adaptive weights, i.e. adjusting the weights between perspectives based on adaptive weight factors, the model's performance was optimized in all four specific tasks, with the average success rate increasing from 0.608 to 0.638.

[0206] Compared to adaptive pooling methods, the hierarchical pruning method of this disclosure has a significant advantage in preserving feature uniqueness. This further verifies the technical advancement of the feature simplification mechanism based on importance prediction described in this disclosure for refined operation tasks.

[0207] During testing and verification, the inventor, such as Figure 11 As shown, in the 0%-70% pruning rate stage, as the pruning rate increases, the success rates of the three tasks—picking up bottles, hitting with a hammer, and placing shoes—not only do not decrease, but also show a steady upward or leveling-off trend; the success rate reaches its highest point when the pruning rate reaches about 60%-70%; when the pruning rate is too high, exceeding 70%, the success rates of each task drop significantly due to the loss of too much key visual information.

[0208] pass Figure 11 It can be confirmed that appropriate pruning, such as 60%, can improve performance by removing background noise or other redundant information, allowing the model to focus more on task-related core features. Furthermore, this experiment demonstrates that BFA++ can effectively improve the robot's operational accuracy while significantly reducing computational load.

[0209] By introducing a robot operation task control model based on a deep reinforcement learning visual language action model, the online lightweight importance prediction network can be guided to learn attention allocation rules. This enables the model to intelligently perceive regions by first selecting the viewpoint and then focusing on the area during the inference phase. Combined with hierarchical pruning, multi-view visual data is compressed, reducing computational load and improving the robot's accuracy and success rate in complex, unstructured environments by eliminating background noise interference.

[0210] In an exemplary embodiment of this disclosure, a robot operation task control device based on a visual language action model is also provided. (See reference...) Figure 12 As shown, the robot operation task control device 1200 based on visual language action model includes a first prediction module 1201, an extraction module 1202, a second prediction module 1203, a first fusion module 1204, and a strategy output module 1205, wherein: The first prediction module 1201 is used to predict the inter-view weights of the corresponding visual image data based on the global feature vectors corresponding to the visual image data from multiple perspectives of the robot, using a pre-trained inter-view importance prediction network. Extraction module 1202 is used to extract features from each image region in each visual image data to obtain multiple local feature vectors corresponding to the visual image data. The second prediction module 1203 is used to predict the view weight of the corresponding image region based on the local feature vector and using a pre-trained view importance prediction network. The first fusion module 1204 is used to combine the interview weights and the intraview weights of each image region to fuse the feature vectors of the preserved regions in each visual image data to obtain a fused feature vector. The strategy output module 1205 is used to input the fused feature vector into the robot operation task control model pre-trained based on the visual language action model algorithm to obtain operation task control instructions.

[0211] The specific details of each module in the above-mentioned robot operation task control device based on visual language action model have been described in detail in the corresponding robot operation task control method based on visual language action model, so they will not be repeated here.

[0212] In an exemplary embodiment of this disclosure, a training device for a robot operation task control model based on a visual language action model is also provided. (See reference...) Figure 13 As shown, the training device 1300 for the robot operation task control model based on the visual language action model includes a determination module 1301, a third prediction module 1302, a second fusion module 1303, and a policy training module 1304, wherein: The determination module 1301 is used to determine the ground truth labels of importance between viewpoints and importance within viewpoints based on the training data; wherein, the training data includes visual image data of multiple viewpoints during the robot's task execution, robot action labels corresponding to each visual image data, and robot state description information; The third prediction module 1302 is used to determine the prediction weights between viewpoints based on the global feature vector using the inter-view importance prediction network, and to determine the prediction weights within the viewpoint based on the local feature vector using the intra-view importance prediction network. The second fusion module 1303 is used to combine the inter-view prediction weights and intra-view prediction weights to perform weighted fusion of local feature vectors, and to obtain the predicted action based on the fused feature vector. The policy training module 1304 is used to update the parameters of the inter-view importance prediction network, the intra-view importance prediction network, and the policy network based on the first auxiliary loss between the inter-view prediction weights and the ground truth labels of inter-view importance, the second auxiliary loss between the intra-view prediction weights and the ground truth labels of intra-view importance, and the action prediction loss between the predicted action and the robot action label.

[0213] The specific details of each module in the training device for the robot operation task control model based on the visual language action model have been described in detail in the corresponding training method for the robot operation task control model based on the visual language action model, so they will not be repeated here.

[0214] In an exemplary embodiment of this disclosure, a robot is also provided. The robot includes a processor and a memory, the memory storing computer-readable instructions that, when executed by the processor, implement the aforementioned method. The robot includes any one of the following: legged robot, quadrupedal robot, bipedal robot, wheeled robot, wheel-legged robot, quadrupedal robot, humanoid robot, cleaning robot, transport robot, mobile robot, and robotic arm.

[0215] refer to Figure 14 The diagram shows a humanoid robot, including a robotic arm 1400.

[0216] refer to Figure 15 As shown, an electronic device capable of implementing the above method is also provided. The electronic device 1500 includes a processor 1501 and a memory 1502. The memory 1502 stores computer-readable instructions, which, when executed by the processor 1501, implement the method of this disclosure.

[0217] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, having stored thereon computer program code instructions that, when invoked by a robot's processor, cause the robot to perform the method as described in the embodiments.

[0218] refer to Figure 16 As shown, a program product 1600 for implementing the above-described method according to an embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0219] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0220] Finally, the above preferred embodiments are only used to illustrate the technical solutions of this disclosure and are not restrictive. Although this disclosure has been described in detail, those skilled in the art should understand that changes in form and detail can be made without departing from the scope defined by the claims of this disclosure. The dimensions in the drawings are not related to the specific physical object, and the physical object dimensions can be arbitrarily changed.

Claims

1. A robot operation task control method based on a visual language action model, characterized in that, The method includes: Based on the global feature vectors corresponding to the visual image data from multiple perspectives of the robot, a pre-trained inter-view importance prediction network is used to predict the inter-view weights of the corresponding visual image data. For each image region in each of the visual image data, feature extraction is performed to obtain multiple local feature vectors corresponding to the visual image data. Based on the local feature vectors, a pre-trained in-view importance prediction network is used to predict the in-view weights of the corresponding image regions. By combining the interview weights and the intraview weights of each image region, the feature vectors of the preserved regions in each visual image data are fused to obtain a fused feature vector; The fused feature vector is input into a robot operation task control strategy network based on a visual language action model to obtain operation task control instructions.

2. The robot operation task control method based on visual language action model according to claim 1, characterized in that, The step of combining the inter-viewpoint weights and the intra-viewpoint weights of each image region to fuse the feature vectors of the preserved regions in each visual image data to obtain a fused feature vector includes: Based on the inter-viewpoint weights and intra-viewpoint weights, the image regions in each visual image data are pruned to obtain the retained regions in each visual image data. The feature vectors of the preserved regions in each of the aforementioned visual image data are fused to obtain the fused feature vector.

3. The robot operation task control method based on visual language action model according to claim 1, characterized in that, The step of combining the inter-viewpoint weights and the intra-viewpoint weights of each image region to fuse the feature vectors of the preserved regions in each visual image data to obtain a fused feature vector includes: The interview weights of each visual image data are multiplied by the intraview weights of the corresponding image regions to obtain the global importance score of each image region. Image regions whose global importance scores are greater than a preset score threshold are identified as the reserved regions; The local feature vectors of each of the reserved regions are weighted according to the global importance score to obtain the weighted feature vectors of each reserved region. The weighted feature vectors of each preserved region in each visual image data are concatenated to obtain the fused feature vector.

4. The robot operation task control method based on visual language action model according to claim 2, characterized in that, The step of pruning the image regions in each of the visual image data based on the inter-viewpoint weights and the intra-viewpoint weights to obtain the retained regions in each of the visual image data includes: Obtain the interview weights corresponding to each visual image data, and determine the visual image data whose interview weights are greater than a preset interview threshold as candidate view data. The image regions in each candidate viewpoint data are sorted based on their corresponding viewpoint weights to obtain the sorting results; Based on the sorting results, the reserved regions in each of the candidate viewpoint data are determined, and the image regions of the reserved regions are selected from each of the candidate viewpoint data.

5. The robot operation task control method based on visual language action model according to claim 2, characterized in that, The step of pruning the image regions in each of the visual image data based on the inter-viewpoint weights and the intra-viewpoint weights to obtain the retained regions in each of the visual image data includes: Using the in-view weights of the image regions in each of the visual image data, local pruning is performed on each of the image regions to obtain the primary preserved regions of each of the visual image data. Based on the inter-viewpoint weights and the intra-viewpoint weights, calculate the global importance score of each image region in each of the primary retention regions; Based on the global importance score of each image region in each of the primary preserved regions, global pruning is performed on each of the primary preserved regions in all the visual image data to obtain the preserved regions in each of the visual image data.

6. The robot operation task control method based on visual language action model according to claim 1, characterized in that, The step of predicting the view weights of the corresponding image regions based on the local feature vectors using a pre-trained in-view importance prediction network includes: The local feature vectors are initially scored using the in-view importance prediction network to obtain the original importance score of the image region to be predicted; Calculate the spatial distance between the image region to be predicted and other image regions in the visual image data; wherein, the other image regions are all image regions in the visual image data other than the image region to be predicted; The original importance score is then smoothed and weighted using the spatial distance to obtain the processed importance score. The importance scores of each image region in the visual image data are normalized to obtain the in-view weight of the image region to be predicted.

7. The robot operation task control method based on visual language action model according to claim 6, characterized in that, The step of smoothing and weighting the original importance score using the spatial distance to obtain the processed importance score includes: The ratio of the original importance score of the other image regions to the corresponding spatial distance is determined as the distance attenuation term for the other image regions; The correction value is determined based on the weighted sum of the distance attenuation terms corresponding to the other image regions; The sum of the corrected value and the original importance score is determined as the processed importance score.

8. The robot operation task control method based on visual language action model according to claim 1, characterized in that, The step of predicting the viewpoint weights of the corresponding visual image data based on each of the global feature vectors and using a pre-trained view importance prediction network includes: The global feature vectors of each of the aforementioned visual image data are concatenated to construct a multi-view context vector; The multi-view context vector is input into the inter-view importance prediction network, and the inter-view importance prediction network outputs the initial prediction value. The initial predicted values ​​are normalized to obtain the viewpoint weights of each visual image data.

9. The robot operation task control method based on visual language action model according to claim 1, characterized in that, The step of extracting features from each image region in each of the visual image data to obtain multiple local feature vectors corresponding to the visual image data includes: The sequence of each image region in each of the aforementioned visual image data is input into a pre-trained visual encoder, and the self-attention mechanism of the pre-trained visual encoder is used to process each of the aforementioned image regions to generate a feature sequence including category tokens and image block tokens. The image block tokens are separated from the feature sequence, and the local feature vectors corresponding to the visual image data are determined based on the image block tokens corresponding to each image region.

10. A training method for a robot operation task control model based on a visual language action model, characterized in that, The model includes at least an inter-view importance prediction network, an intra-view importance prediction network, and a robot operation task control policy network; the method includes: The ground truth labels for importance between viewpoints and importance within viewpoints are determined based on the training data; wherein, the training data includes visual image data from multiple viewpoints during the robot's task execution, robot action labels corresponding to each visual image data, and robot state description information; The inter-view importance prediction network is used to determine the inter-view prediction weights based on the global feature vector of the visual image data, and the intra-view importance prediction network is used to determine the intra-view prediction weights based on the local feature vector of the visual image data. The local feature vector is weighted and fused by combining the inter-view prediction weights and the intra-view prediction weights, and the predicted action is obtained by using the robot operation task control strategy network based on the fused feature vector. Based on the first auxiliary loss between the inter-view prediction weights and the ground truth labels of inter-view importance, the second auxiliary loss between the intra-view prediction weights and the ground truth labels of intra-view importance, and the action prediction loss between the predicted action and the robot action label, the parameters of the inter-view importance prediction network, the intra-view importance prediction network, and the policy network are updated respectively.

11. The training method for a robot operation task control model based on a visual language action model according to claim 10, characterized in that, The step of determining ground truth labels for inter-view importance and intra-view importance based on training data includes: Based on the robot state description information corresponding to each visual image data, determine the true value label of the importance between the viewpoints of each visual image data. Based on the correspondence between each image region in each visual image data and the task entity, a ground truth label for the visual image data within the viewpoint is generated.

12. The training method for the robot operation task control model based on the visual language action model according to claim 11, characterized in that, The step of determining the ground truth labels of the importance between viewpoints of each visual image data based on the robot state description information corresponding to each visual image data includes: The current operational stage of each robot is determined based on the state description information of each robot; If the operation stage is the object interaction stage, then the viewpoint importance truth labels of the visual image data corresponding to the main viewpoint and the visual image data containing at least one robot arm viewpoint in each of the visual image data are marked as the first value. If the operation phase is a non-interactive phase, then the ground truth label of the importance between the viewpoints of the visual image data corresponding to the main viewpoint in each of the visual image data is marked as the first value, and the ground truth label of the importance of the visual image data containing the robot arm's viewpoint is marked as the second value.

13. The training method for the robot operation task control model based on the visual language action model according to claim 11, characterized in that, The step of generating ground truth labels for the visual importance of each visual image data based on the correspondence between each image region in each visual image data and the task entity includes: Obtain the target region range of the task entity in each of the aforementioned visual image data; The pixel in each of the visual image data that is within the corresponding target region is determined as the first pixel, and the importance truth value label of the first pixel within the viewpoint is marked as the first value. Pixels in each of the visual image data that are not within the corresponding target region are identified as second pixels, and the ground truth label of the importance of the second pixel within its viewpoint is marked as a second value.

14. The training method for the robot operation task control model based on a visual language action model according to claim 11, characterized in that, The task entity includes at least a robotic arm actuator and the object being manipulated.

15. A robot operation task control device based on a visual language action model, characterized in that, include: The first prediction module is used to predict the inter-view weights of the visual image data based on the global feature vectors corresponding to the visual image data from multiple perspectives of the robot, using a pre-trained inter-view importance prediction network. The extraction module is used to extract features from each image region in each of the visual image data to obtain multiple local feature vectors corresponding to the visual image data. The second prediction module is used to predict the view weight of the corresponding image region based on the local feature vector using a pre-trained view importance prediction network. The first fusion module is used to combine the interview weights and the intraview weights of each image region to fuse the feature vectors of the reserved regions in each visual image data to obtain a fused feature vector. The strategy output module is used to input the fused feature vector into the robot operation task control strategy network based on the visual language action model to obtain operation task control instructions.

16. A training device for a robot operation task control model based on a visual language action model, characterized in that, include: The determination module is used to determine the ground truth labels of importance between viewpoints and importance within viewpoints based on the training data; wherein, the training data includes visual image data of multiple viewpoints during the robot's task execution, robot action labels corresponding to each visual image data, and robot state description information; The third prediction module is used to determine the prediction weights between viewpoints based on the global feature vector using the inter-view importance prediction network, and to determine the prediction weights within the viewpoint based on the local feature vector using the intra-view importance prediction network. The second fusion module is used to perform weighted fusion of the local feature vector by combining the inter-view prediction weights and the intra-view prediction weights, and to obtain the predicted action based on the fused feature vector. The policy training module is used to update the parameters of the inter-view importance prediction network, the intra-view importance prediction network, and the policy network based on the first auxiliary loss between the inter-view prediction weights and the ground truth labels of inter-view importance, the second auxiliary loss between the intra-view prediction weights and the ground truth labels of intra-view importance, and the action prediction loss between the predicted action and the robot action label.

17. An electronic device, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 14.

18. A robot, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 14.

19. The robot according to claim 18, characterized in that, The robot includes any one of the following: legged robot, quadruped robot, bipedal robot, wheeled robot, wheel-legged robot, quadrupedal robot, humanoid robot, cleaning robot, transport robot, mobile robot, and robotic arm.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program code instructions that, when invoked by the robot's processor, cause the robot to perform the method as described in any one of claims 1 to 14.

Images