Multi-view robot manipulation method and system based on differential attention

Abstract

The application discloses a multi-view mechanical arm operation method and system based on differential attention, and belongs to the field of mechanical arm operation, comprising the following steps: acquiring a mechanical arm operation scene observation point cloud and rendering the same into a multi-view orthogonal projection image, extracting image block features in blocks, and simultaneously extracting natural language sequence feature; calculating cross attention through a multi-head differential attention mechanism, calculating self-attention through the multi-head differential attention mechanism by using the cross attention; inputting the self-attention output feature into an action prediction network to obtain a next operation action of the mechanical arm; the multi-head differential attention mechanism: linearly mapping and splitting features to obtain multiple sets of query, key and value features, calculating differential mode and common mode attention values by each attention head, and obtaining the output result of the multi-head differential attention mechanism after normalization, weighting and splicing mapping of overall optimization. The differential attention mechanism proposed in the application constructs a new multi-view-based mechanical arm operation method, the training duration is greatly reduced, and excellent performance is achieved on high-precision operation tasks.

Description

Technical Field

[0001] This invention belongs to the field of robotic arm operation, specifically relating to a multi-view robotic arm operation method and system based on differential attention. Background Technology

[0002] Achieving robot manipulation in unstructured 3D environments requires both accurate reasoning capabilities and strong adaptability. In recent years, multi-view 3D representation methods have gradually become a powerful paradigm because they provide rich visual cues, crucial for accurate 3D understanding and task execution. Compared to voxel representation, methods based on 3D multi-view representation (such as the Robotic View Transformer, RVT) have achieved superior performance while significantly reducing training costs, demonstrating their potential in robotic arm manipulation. However, these methods still face several shortcomings in practical applications:

[0003] Existing methods typically rely on standard attention mechanisms to locate task-relevant regions. However, they often suffer from "attention drift," where the model's focus shifts from meaningful cues to redundant or noisy content (e.g., cluttered backgrounds, occlusions, or repetitive textures). This is because the robot's perspective contains a large amount of redundant and task-irrelevant information, and standard attention mechanisms have limited ability to suppress irrelevant signals, leading to a decrease in reasoning and action accuracy.

[0004] Existing robotic arm manipulation methods based on 3D multi-view representation perform poorly on high-precision tasks (such as socketing tasks), and there is a lack of a reliable method that can meet the requirements of high-precision operation.

[0005] The most advanced robotic arm operation methods based on 3D multi-view representation currently have low training efficiency, typically requiring around 100,000 training steps, resulting in high training time costs.

[0006] The above analysis shows that there is an urgent need for an attention mechanism that can effectively capture task-related information, and based on this mechanism, an efficient robotic arm operation method should be designed to achieve high-precision and high-reliability operation. Summary of the Invention

[0007] The purpose of this invention is to overcome the problems of low training efficiency, low accuracy, and poor reliability of existing multi-view robotic arm operation methods, and to propose a multi-view robotic arm operation method and system based on differential attention.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a multi-view robotic arm operation method based on differential attention, comprising the following steps:

[0010] Acquire the observation point cloud of the robotic arm operation scene and render the observation point cloud into a multi-view orthogonal projection image;

[0011] Multi-view orthogonal projection images are segmented and preprocessed to extract multi-view image patch features, while natural language sequence features of the target operation task are also extracted.

[0012] Calculate the cross-attention between multi-view image patch features and natural language sequence features, and perform self-attention calculation on the output features of the cross-attention to obtain the output features of the self-attention calculation;

[0013] The output features calculated by self-attention are input into the action prediction network to obtain the next action of the robotic arm.

[0014] Both cross-attention and self-attention calculations employ a multi-head differential attention mechanism, specifically:

[0015] The input features are linearly mapped to obtain the mapped features. The mapped features are then split to obtain multiple sets of query features, key features, and value features. Each set of query features, key features, and value features corresponds to an attention head.

[0016] For each attention head, two sets of basic attention values ​​are calculated based on the corresponding query features, key features, and value features. The differential attention value and common attention value of the current attention head are then calculated based on the two sets of basic attention values.

[0017] By using the loss function, the differential attention value and common attention value of all attention heads are optimized as a whole to obtain the optimized differential attention value of each attention head;

[0018] The differential attention values ​​optimized for each attention head are normalized and weighted by coefficients. The processing results of all attention heads are then concatenated and linearly mapped to obtain the output of the multi-head differential attention mechanism.

[0019] Furthermore, the number of multi-view orthogonal projection images is three, and the three orthogonal projection images are obtained by rendering the observation point cloud through three mutually perpendicular orthogonal directions;

[0020] Multi-view image patch features are represented by embedding, and the embedding is represented by a pre-trained visual representation encoder for multi-view orthogonal projection images.

[0021] Furthermore, the linear mapping of the input features specifically involves mapping the input features through three independent linear layers, and the mapped features are composed of the output features of the three linear layers.

[0022] Furthermore, the two sets of basic attention values ​​include a first basic attention value and a second basic attention value;

[0023] Both the first and second basic attention values ​​are calculated based on the standard attention calculation formula, which is a weighted sum of attention weights and corresponding value features.

[0024] The differential attention value is calculated by the weighted difference between the first and second basic attention values, and the common attention value is calculated by the arithmetic mean of the first and second basic attention values.

[0025] Furthermore, the loss function is shown in the following equation:

[0026]

[0027]

[0028] in This represents the common-mode rejection ratio loss of the entire differential attention module. This represents the common-mode attention score of the nth layer attention block. This represents the differential attention score of the nth layer attention block. Indicates the number of attention blocks; This indicates the calculation of the common-mode suppression ratio loss value for the nth attention block.

[0029] Furthermore, the specific method for block preprocessing of multi-view orthogonal projection images is as follows:

[0030] The orthogonal projection image of each viewpoint is evenly divided into blocks according to a preset size. The features of each image block are extracted and the dimensions are unified to obtain the multi-view image block features.

[0031] The method for extracting natural language sequence features is as follows:

[0032] The natural language description of the target operation task is encoded by a pre-trained language model, and the output is a fixed-dimensional feature vector as the natural language sequence feature.

[0033] Furthermore, the motion prediction network is a fully connected neural network or a convolutional neural network, and the output of the motion prediction network is the angle control quantity of each joint of the robotic arm or the pose control quantity of the end effector.

[0034] Secondly, this invention provides a multi-view robotic arm operating system based on differential attention, including:

[0035] The point cloud rendering module is used to acquire the observation point cloud of the robotic arm operation scene and render the observation point cloud into a multi-view orthogonal projection image.

[0036] The feature extraction module is used to perform block preprocessing on multi-view orthogonal projection images to extract multi-view image block features, and at the same time extract natural language sequence features corresponding to the target operation task.

[0037] The attention calculation module is used to calculate the cross-attention between multi-view image patch features and natural language sequence features, and to perform self-attention calculation on the output features of the cross-attention. Both the cross-attention calculation and the self-attention calculation adopt a multi-head differential attention mechanism, specifically:

[0038] The input features are linearly mapped to obtain the mapped features. The mapped features are then split to obtain multiple sets of query features, key features, and value features. Each set of query features, key features, and value features corresponds to an attention head.

[0039] For each attention head, two sets of basic attention values ​​are calculated based on the corresponding query features, key features, and value features. The differential attention value and common attention value of the current attention head are then calculated based on the two sets of basic attention values.

[0040] By using the loss function, the differential attention value and common attention value of all attention heads are optimized as a whole to obtain the optimized differential attention value of each attention head;

[0041] The differential attention values ​​optimized for each attention head are normalized and weighted by coefficients. The processing results of all attention heads are concatenated and linearly mapped to obtain the output of the multi-head differential attention mechanism.

[0042] The robotic arm motion prediction module is used to input the output features calculated by self-attention into the motion prediction network to obtain the next operation motion of the robotic arm.

[0043] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a multi-view robotic arm operation method based on differential attention.

[0044] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a multi-view robotic arm operation method based on differential attention.

[0045] Compared with the prior art, the present invention has the following beneficial technical effects:

[0046] This invention proposes a multi-view robotic arm manipulation method based on differential attention. Existing attention calculations in robotic arm manipulation strategies typically rely on conventional attention mechanisms. However, due to the complexity of robotic arm manipulation scenarios, this attention calculation method is easily affected by interference from non-target areas of interest, leading to attention drift. Inspired by differential amplifiers, this invention proposes a differential attention mechanism that enhances attention to the target area while reducing attention to non-target areas, improving the model's ability to capture high signal-to-noise ratio scene information and solving this problem. Existing multi-view robotic arm manipulation methods suffer from long training cycles, primarily due to the model's insufficient ability to efficiently capture information relevant to the target task. Therefore, a novel multi-view robotic arm manipulation method is constructed based on the proposed differential attention mechanism, significantly reducing training time and demonstrating excellent performance on high-precision manipulation tasks. Experimental results show that this invention achieves the best average success rate across 18 tasks on the RLBench benchmark while halving the training step size. Existing multi-view robotic arm manipulation methods perform poorly on high-precision tasks (e.g., socketing tasks). Thanks to the effectiveness of the differential attention mechanism in capturing high signal-to-noise ratio information, and its modular design that allows for easy combination with other modules, this invention combines the differential attention mechanism with existing pre-trained visual representation encoders (e.g., the SAM2 encoder) in image block preprocessing, significantly improving the success rate of robotic arms in high-precision tasks. Attached Figure Description

[0047] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of the invention in any way. Furthermore, the shapes and proportions of the components in the drawings are merely illustrative to aid in understanding the invention and do not specifically limit the shapes and proportions of the components of the invention.

[0048] In the attached diagram:

[0049] Figure 1 This is a flowchart of the multi-view robotic arm operation method based on differential attention according to the present invention.

[0050] Figure 2 This is a simplified structural diagram of the multi-view robotic arm operating system based on differential attention, as described in this invention.

[0051] Figure 3 This is a schematic diagram of an electronic device for the multi-view robotic arm operation method based on differential attention according to the present invention.

[0052] Figure 4 This is a schematic diagram illustrating the implementation steps of the differential attention mechanism inspired by differential amplifiers in an embodiment of the present invention.

[0053] Figure 5This is a schematic diagram illustrating the implementation steps of the multi-view robotic arm operation method based on differential attention mechanism in an embodiment of the present invention.

[0054] Figure 6 This is a schematic diagram illustrating the visualization and quantification process in an embodiment of the present invention.

[0055] Figure 7 This is a schematic diagram showing the comparative experimental results of the method in this embodiment of the invention with other methods on 18 tasks in the RLBench benchmark. Detailed Implementation

[0056] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0057] Example 1

[0058] See Figure 1 A multi-view robotic arm operation method based on differential attention includes the following steps:

[0059] The system acquires observation point clouds of the robotic arm's operation scene and renders them into multi-view orthogonal projection images. It then performs block preprocessing on the multi-view orthogonal projection images to extract multi-view image patch features, while simultaneously extracting natural language sequence features of the target operation task. The system calculates cross-attention between the multi-view image patch features and the natural language sequence features, and performs self-attention calculation on the output features of the cross-attention calculation to obtain the output features of the self-attention calculation. Finally, it inputs the output features of the self-attention calculation into an action prediction network to obtain the next operation action of the robotic arm.

[0060] Both cross-attention and self-attention calculations employ a multi-head differential attention mechanism, specifically:

[0061] The input features are linearly mapped to obtain mapped features. These mapped features are then split into multiple sets of query features, key features, and value features, each corresponding to an attention head. For each attention head, two sets of basic attention values ​​are calculated based on the corresponding query features, key features, and value features. Based on these two sets of basic attention values, the differential attention value and common attention value of the current attention head are calculated. Using a loss function, the differential attention values ​​and common attention values ​​of all attention heads are optimized as a whole to obtain the optimized differential attention value for each attention head. The optimized differential attention value for each attention head is then normalized and weighted. The processing results of all attention heads are concatenated and linearly mapped to obtain the output of the multi-head differential attention mechanism.

[0062] Example 2

[0063] See Figure 2 A multi-view robotic arm operating system based on differential attention includes:

[0064] The point cloud rendering module is used to acquire the observation point cloud of the robotic arm operation scene and render the observation point cloud into a multi-view orthogonal projection image.

[0065] The feature extraction module is used to perform block preprocessing on multi-view orthogonal projection images to extract multi-view image block features, and at the same time extract natural language sequence features corresponding to the target operation task.

[0066] The attention calculation module is used to calculate the cross-attention between multi-view image patch features and natural language sequence features, and to perform self-attention calculation on the output features of the cross-attention. Both the cross-attention calculation and the self-attention calculation adopt a multi-head differential attention mechanism, specifically:

[0067] The input features are linearly mapped to obtain the mapped features. The mapped features are then split to obtain multiple sets of query features, key features, and value features. Each set of query features, key features, and value features corresponds to an attention head.

[0068] For each attention head, two sets of basic attention values ​​are calculated based on the corresponding query features, key features, and value features. The differential attention value and common attention value of the current attention head are then calculated based on the two sets of basic attention values.

[0069] By using the loss function, the differential attention value and common attention value of all attention heads are optimized as a whole to obtain the optimized differential attention value of each attention head;

[0070] The differential attention values ​​optimized for each attention head are normalized and weighted by coefficients. The processing results of all attention heads are concatenated and linearly mapped to obtain the output of the multi-head differential attention mechanism.

[0071] The robotic arm motion prediction module is used to input the output features calculated by self-attention into the motion prediction network to obtain the next operation motion of the robotic arm.

[0072] Example 3

[0073] See Figure 3 An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a multi-view robotic arm operation method based on differential attention: acquiring an observation point cloud of the robotic arm operation scene and rendering the observation point cloud into a multi-view orthogonal projection image; performing block preprocessing on the multi-view orthogonal projection image to extract multi-view image block features, and simultaneously extracting natural language sequence features of the target operation task; calculating the cross attention between the multi-view image block features and the natural language sequence features, and performing self-attention calculation on the output features of the cross attention to obtain the output features of the self-attention calculation; inputting the output features of the self-attention calculation into an action prediction network to obtain the next operation action of the robotic arm.

[0074] Both cross-attention and self-attention calculations employ a multi-head differential attention mechanism, specifically:

[0075] The input features are linearly mapped to obtain mapped features. These mapped features are then split into multiple sets of query features, key features, and value features, each corresponding to an attention head. For each attention head, two sets of basic attention values ​​are calculated based on the corresponding query features, key features, and value features. Based on these two sets of basic attention values, the differential attention value and common attention value of the current attention head are calculated. Using a loss function, the differential attention values ​​and common attention values ​​of all attention heads are optimized as a whole to obtain the optimized differential attention value for each attention head. The optimized differential attention value for each attention head is then normalized and weighted. The processing results of all attention heads are concatenated and linearly mapped to obtain the output of the multi-head differential attention mechanism.

[0076] Example 4

[0077] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements a multi-view robotic arm operation method based on differential attention: acquiring an observation point cloud of the robotic arm operation scene and rendering the observation point cloud into a multi-view orthogonal projection image; performing block preprocessing on the multi-view orthogonal projection image to extract multi-view image block features, and simultaneously extracting natural language sequence features of the target operation task; calculating the cross attention between the multi-view image block features and the natural language sequence features, and performing self-attention calculation on the output features of the cross attention to obtain the output features of the self-attention calculation; inputting the output features of the self-attention calculation into an action prediction network to obtain the next operation action of the robotic arm;

[0078] Both cross-attention and self-attention calculations employ a multi-head differential attention mechanism, specifically:

[0079] The input features are linearly mapped to obtain mapped features. These mapped features are then split into multiple sets of query features, key features, and value features, each corresponding to an attention head. For each attention head, two sets of basic attention values ​​are calculated based on the corresponding query features, key features, and value features. Based on these two sets of basic attention values, the differential attention value and common attention value of the current attention head are calculated. Using a loss function, the differential attention values ​​and common attention values ​​of all attention heads are optimized as a whole to obtain the optimized differential attention value for each attention head. The optimized differential attention value for each attention head is then normalized and weighted. The processing results of all attention heads are concatenated and linearly mapped to obtain the output of the multi-head differential attention mechanism.

[0080] Example 5

[0081] A multi-view robotic arm operation method based on differential attention, see [link to relevant documentation] Figure 5 The implementation steps of the multi-view robotic arm operation method based on differential attention are as follows:

[0082] S001. First, input the scene observation point cloud of the robotic arm and render it as an orthogonal projection image from three perspectives.

[0083] S002. Perform block preprocessing on the multi-view images to obtain multi-view image block features. Simultaneously, extract natural language sequence features for the target task. The image block embedding representation utilizes a pre-trained visual representation encoder (such as SAM2, Clip, etc.) to represent the input image. Because SAM2 has strong representational capabilities, in... Figure 7 The results of experiments using sam2 are also shown.

[0084] S003. Calculate the cross-attention between image patch features and task description features from multiple perspectives, where the attention mechanism uses a differential attention mechanism.

[0085] S004. Self-attention calculation is performed based on the features output by S003, and the attention mechanism used is the differential attention mechanism.

[0086] S005. Input the features output from S004 into the motion prediction network to predict the next action of the robotic arm.

[0087] See Figure 4 The implementation steps of the differential attention mechanism inspired by differential amplifiers are as follows:

[0088] S1. First, the input X is mapped using three linear layers.

[0089] S2. Then calculate the differential attention and common mode attention.

[0090] S2.1. Divide the three mapping structures in S1 into equal parts according to the sequence length to obtain the input query. , Input key is , and the input value is , .

[0091] S2.2 Calculate the attention of each head according to the standard attention calculation formula. and The attention value is obtained by concatenating the two results. .

[0092] S2.3 Calculate the attention of each head separately according to the standard attention calculation formula. and The attention value is obtained by concatenating the two results. .

[0093] S2.4 Calculate the differential attention value for each head.

[0094] S2.5 Calculate the common-mode attention value for each head.

[0095] S3, Through Optimize multi-head differential attention mechanism

[0096]

[0097]

[0098] in, This represents the common-mode rejection ratio loss of the entire differential attention module. This represents the common-mode attention score of the nth layer attention block. This represents the differential attention score of the nth layer attention block. This indicates the number of attention blocks. This indicates the calculation of the common-mode suppression ratio loss value for the nth attention block.

[0099] S4, Output of Multi-Head Differential Attention

[0100] S4.1 After normalizing the results of S2.4 using LayerNorm, the output of each header is compared with... Multiplication

[0101] S4.2. Concatenate the output results of each head, and then perform Linear layer mapping to obtain the final differential attention calculation result.

[0102] See Figure 6 To qualitatively and quantitatively analyze the information attention levels of differential attention mechanisms and conventional attention mechanisms in the same task scenario, the visualization and quantification process of this invention is as follows:

[0103] S101. For the task of “opening the top drawer”, the black box is the area that the model needs to pay close attention to.

[0104] S102. Draw attention distribution diagrams for the conventional attention mechanism and the differential attention mechanism in the same task scenario. The darker the color inside the black box, the higher the attention level; the lighter the color outside the black box, the higher the attention level. The diagrams show that the differential attention mechanism pays less attention to task-irrelevant information and pays more attention to task-related information compared to the conventional attention mechanism.

[0105] S103. Calculate the mean attention value for the task-relevant region (TRR, inside the black box) and the task-independent region (TIR, outside the black box) under different attention mechanisms. Calculate the TRR-TIR ratio to reflect the signal-to-noise ratio of the information captured by different attention mechanisms.

[0106] Existing multi-view robotic arm manipulation methods suffer from long training cycles, primarily due to the model's insufficient ability to efficiently capture task-related information. To address this, a novel multi-view robotic arm manipulation method is constructed based on the proposed differential attention mechanism. This method achieves top-tier average success rate across 18 tasks on the RLBench benchmark while halving the training step size. Specific experimental results comparing this invention's method with other methods on 18 tasks on the RLBench benchmark are as follows: Figure 7 As shown.

[0107] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0108] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0109] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A multi-view robotic arm operation method based on differential attention, characterized in that, Includes the following steps: Acquire the observation point cloud of the robotic arm operation scene, and render the observation point cloud into a multi-view orthogonal projection image; The multi-view orthogonal projection image is preprocessed into blocks to extract multi-view image block features, and natural language sequence features of the target operation task are extracted simultaneously. Calculate the cross-attention between the multi-view image patch features and the natural language sequence features, and perform self-attention calculation on the output features of the cross-attention to obtain the output features of the self-attention calculation; The output features calculated by the self-attention method are input into the action prediction network to obtain the next action of the robotic arm. The calculation of cross-attention and the calculation of self-attention both employ a multi-head differential attention mechanism, specifically: The input features are linearly mapped to obtain the mapped features. The mapped features are then split to obtain multiple sets of query features, key features, and value features. Each set of query features, key features, and value features corresponds to an attention head. For each attention head, two sets of basic attention values ​​are calculated based on the corresponding query features, key features, and value features. The differential attention value and common attention value of the current attention head are then calculated based on the two sets of basic attention values. By using the loss function, the differential attention value and common attention value of all attention heads are optimized as a whole to obtain the optimized differential attention value of each attention head; The differential attention values ​​optimized for each attention head are normalized and weighted by coefficients. The processing results of all attention heads are then concatenated and linearly mapped to obtain the output of the multi-head differential attention mechanism.

2. The multi-view robotic arm operation method based on differential attention according to claim 1, characterized in that, The number of multi-view orthogonal projection images is three, and the three orthogonal projection images are obtained by rendering the observation point cloud through three mutually perpendicular orthogonal directions; The multi-view image patch features are represented by an embedding representation, which is represented by a pre-trained visual representation encoder on the multi-view orthogonal projection image.

3. The multi-view robotic arm operation method based on differential attention according to claim 1, characterized in that, The linear mapping of the input features specifically involves mapping the input features through three independent linear layers, and the mapped features are composed of the output features of the three linear layers.

4. The multi-view robotic arm operation method based on differential attention according to claim 1, characterized in that, The two sets of basic attention values ​​include a first basic attention value and a second basic attention value; Both the first basic attention value and the second basic attention value are calculated according to the standard attention calculation formula, which is a weighted sum of attention weights and corresponding value features; The differential attention value is calculated by the weighted difference between the first basic attention value and the second basic attention value, and the common attention value is calculated by the arithmetic mean of the first basic attention value and the second basic attention value.

5. The multi-view robotic arm operation method based on differential attention according to claim 1, characterized in that, The loss function is shown in the following equation: in This represents the common-mode rejection ratio loss of the entire differential attention module. This represents the common-mode attention score of the nth layer attention block. This represents the differential attention score of the nth layer attention block. Indicates the number of attention blocks; This indicates the calculation of the common-mode suppression ratio loss value for the nth attention block.

6. The multi-view robotic arm operation method based on differential attention according to claim 1, characterized in that, The specific method for performing block-based preprocessing on the multi-view orthogonal projection image is as follows: The orthogonal projection image of each viewpoint is evenly divided into blocks according to a preset size, the features of each image block are extracted and the dimensions are unified to obtain the multi-view image block features. The method for extracting the natural language sequence features is as follows: The natural language description of the target operation task is encoded by a pre-trained language model, and a fixed-dimensional feature vector is output as the natural language sequence feature.

7. The multi-view robotic arm operation method based on differential attention according to claim 1, characterized in that, The motion prediction network is a fully connected neural network or a convolutional neural network, and the output of the motion prediction network is the angle control quantity of each joint of the robotic arm or the pose control quantity of the end effector.

8. A multi-view robotic arm operating system based on differential attention, characterized in that, include: The point cloud rendering module is used to acquire the observation point cloud of the robotic arm operation scene and render the observation point cloud into a multi-view orthogonal projection image. The feature extraction module is used to perform block preprocessing on the multi-view orthogonal projection image to extract multi-view image block features, and at the same time extract the natural language sequence features corresponding to the target operation task. The attention calculation module is used to calculate the cross-attention between the multi-view image patch features and the natural language sequence features, and to perform self-attention calculation on the output features of the cross-attention; both the calculation of the cross-attention and the calculation of the self-attention adopt a multi-head differential attention mechanism, specifically: The input features are linearly mapped to obtain the mapped features. The mapped features are then split to obtain multiple sets of query features, key features, and value features. Each set of query features, key features, and value features corresponds to an attention head. For each attention head, two sets of basic attention values ​​are calculated based on the corresponding query features, key features, and value features. The differential attention value and common attention value of the current attention head are calculated based on the two sets of basic attention values. By using the loss function, the differential attention value and common attention value of all attention heads are optimized as a whole to obtain the optimized differential attention value of each attention head; The optimized differential attention value of each attention head is normalized and weighted by coefficients. The processing results of all attention heads are concatenated and linearly mapped to obtain the output result of the multi-head differential attention mechanism. The robotic arm motion prediction module is used to input the output features of the self-attention calculation into the motion prediction network to obtain the next operation motion of the robotic arm.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the multi-view robotic arm operation method based on differential attention as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the multi-view robotic arm operation method based on differential attention as described in any one of claims 1-7.

Images