Multi-view representation fusion method applied to mechanical arm

Abstract

The invention discloses a multi-view representation fusion method applied to a mechanical arm, and belongs to the technical field of reinforcement learning. The implementation method comprises the following steps: 1, capturing and forming an observation image; 2, a mechanical arm multi-view fusion model is constructed, and original multi-view representation is introduced into state representation and position coding to form multi-view position state representation; state embedding is formed for multi-view position state representation through a multi-head self-attention fusion layer, and multi-view representation fusion is optimized by using double simulation metrics, masks and reconstruction loss; 3, training the multi-view fusion model of the mechanical arm by using the joint optimization target; 4, the multi-view set is input into the trained mechanical arm multi-view fusion model, and multi-view fusion representation is output; the multi-view fusion representation is used for mechanical arm task control; compared with the prior art, under the scene of view missing and noise interference, the fusion capability of multi-view representation of the mechanical arm is enhanced, and the task execution efficiency of reinforcement learning on the mechanical arm is improved.

Description

TECHNICAL FIELD

[0001] The application relates to a multi-view representation fusion method applied to a mechanical arm and belongs to the technical field of reinforcement learning and is applied to the image perception and control task direction of the mechanical arm. BACKGROUND

[0002] In a robot operation task, perceiving fine three-dimensional structure information is crucial for improving the grasping efficiency and interaction stability. Usually, three-dimensional sensors are relied on to obtain the spatial information of a scene, but in actual deployment, the high cost and complex configuration of these devices limit their wide application. In recent years, multi-view reinforcement learning (MVRL) has gradually attracted attention as a more economical and flexible alternative. By collecting two-dimensional images from different perspectives and fusing the spatiotemporal information therein, MVRL is expected to achieve depth understanding of the scene and optimization of the action strategy at a lower cost.

[0003] Although MVRL shows great potential in improving the perception ability and reducing the system complexity, the current method still faces two core bottlenecks. First, the multi-view observation data has high redundancy and complex dimensions, and lacks an efficient and task-aware representation extraction mechanism, which easily leads to noise interference in the strategy learning process and reduces the control accuracy. Second, in a real operating environment, factors such as view occlusion, hardware failure or limited field of view can easily lead to missing views or incomplete observations, and existing methods mostly assume complete observations and lack a robust mechanism to deal with input missing or disturbance. In addition, although existing multi-view fusion methods use 3D key points, specific view variational autoencoders or cross-view attention mechanisms to realize multi-view information fusion, they often ignore task-related modeling, making it difficult to effectively distinguish between key and non-key view features and affecting the final strategy performance.

[0004] Therefore, how to enhance the fusion capability of multi-view representation of the mechanical arm in the scenario of missing views and noise interference, and further improve the efficiency of reinforcement learning in the execution of the mechanical arm task has become a key problem that needs to be broken through at present. SUMMARY

[0005] The purpose of the present application is to solve the technical problem of enhancing the fusion capability of multi-view representation of the mechanical arm in the scenario of missing views and noise interference, and further improving the efficiency of reinforcement learning in the execution of the mechanical arm task, and a multi-view representation fusion method applied to a mechanical arm is proposed.

[0006] The purpose of the present application is achieved by the following technical solutions:

[0007] Step 1: Real-time capture images using the multi-view camera of the mechanical arm to form observation images;

[0008] Step 2: Constructing a multi-view fusion model of the mechanical arm, introducing the original multi-view representation into the state representation and position encoding to form a multi-view position state representation; passing the multi-view position state representation through a multi-head self-attention fusion layer to form a state embedding, and optimizing the multi-view representation fusion by using a double analog metric and a mask and reconstruction loss;

[0009] Step 2.1: Constructing an original multi-view set as shown in formula (1) using the observed image;

[0010]

[0011] wherein o i represents the observed image of the i-th view, H, W, and C represent the height, width, and channel number of the image, respectively;

[0012] Step 2.2: Constructing a convolutional encoder and a multi-head self-attention fusion module, using a convolutional neural network to encode the original multi-view set, mapping the original view to an embedding vector, and forming an original multi-view representation as shown in formula (2);

[0013] x i ={x 1 ,x 2 ,...,x k} (2)

[0014] wherein, d is the embedding dimension of the hidden space;

[0015] Step 2.3: Introducing the original multi-view representation into a state representation to be trained and a one-dimensional position encoding E pos , forming a multi-view position state representation as shown in formula (3);

[0016]

[0017] Step 2.4: Constructing a multi-head self-attention fusion layer, passing the multi-view position state representation through the fusion layer as shown in formulas (4) and (5) to form a state embedding;

[0018] z′ l =MHSA(LN(z l-1 ))+z l-1 ,l=1....L (4)

[0019] z l =MLP(LN(z′ l ))+z′ l· l=1....L (5)

[0020] wherein, MHSA(·) represents a multi-head self-attention layer; LN(·) represents a normalization processing on the features; MLP(·) represents a multi-layer perceptron; L represents the number of multi-head self-attention layers; and φ as a state embedding;

[0021] Step 2.5: optimizing the multi-view representation fusion by using a dual-analog metric;

[0022] Step 2.5.1: randomly selecting two multi-views from the original multi-view set

[0023] Step 2.5.2: obtaining the similarity metric distance of the two multi-views in the same action as the equivalent state representation by using a manner as shown in equation (6); and the same probability distribution under the same action as the equivalent state representation as a judgment condition;

[0024]

[0025] wherein, φ ω (·) represents a convolutional encoder and a multi-head self-attention fusion module; and ω represents a model parameter;

[0026] Step 2.5.3: optimizing the φ ω (·) fusion module by using a manner as shown in equation (7);

[0027]

[0028] wherein, represents a dual-analog fusion loss; and γ represents a weight coefficient; a dynamics model in the latent space, is a replay buffer; a represents a task action; s i′ ,s j′ represents a state representation of the latent space at the next moment;

[0029] Step 2.6: optimizing the multi-view representation fusion by using a mask and a reconstruction loss;

[0030] Step 2.6.1: randomly performing pixel covering on the original multi-view set to form a mask observation sequence

[0031] Step 2.6.2: passing the mask observation sequence through a convolutional encoder and a multi-head self-attention fusion module to form a masked state embedding

[0032] Step 2.6.3: constructing a prediction head model, and obtaining a reconstructed state embedding by using the prediction head model

[0033] Step 2.6.4: reconstruct the loss with the mask as shown in formula (8) to predict the head model and ω (·) the fusion module is optimized at the same time;

[0034]

[0035] Step 2.7: establish a joint optimization target as shown in formula (9);

[0036]

[0037] wherein, λ represents the weighting coefficient of the reconstruction loss;

[0038] Step 3: train the multi-view fusion model of the robot arm by using the joint optimization target;

[0039] Step 4: input the multi-view set into the trained multi-view fusion model of the robot arm, and output the multi-view fusion representation; and use the multi-view fusion representation for robot arm task control;

[0040] Compared with the prior art, the present application has the following beneficial effects:

[0041] 1. The present application proposes a novel multi-view representation learning and fusion algorithm, which is the first time to integrate dual analog measurement into the multi-view reinforcement learning framework. This integration can extract task-related representations from redundant multi-view observations. In addition, the present application can be transferred to any existing reinforcement learning algorithm with minimal adjustments.

[0042] 2. The present application proposes a mask and latent reconstruction auxiliary task based on multi-view, which significantly enhances multi-view representation fusion. By introducing a learnable mask label, the robustness to missing views is improved.

[0043] 3. A large number of experiments on motion and robot operation tasks show that the present application is superior to existing multi-view reinforcement learning algorithms, and can always achieve excellent performance even in challenging scenarios with missing or noisy views. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 a schematic diagram of the framework of the present application;

[0045] Figure 2 a performance comparison result diagram of five robot operation tasks in the Meta-World environment. DETAILED DESCRIPTION

[0046] In order to better illustrate the purpose and advantages of the present application, the content of the application is further described below in combination with the drawings and examples.

[0047] EMBODIMENT

[0048] As Figure 1 shown, the present example discloses a multi-view representation fusion-based robot arm reinforcement learning control method, and the specific implementation steps are as follows:

[0049] Step 1: Real-time capture of images using multi-view cameras of the robot arm to form observation images;

[0050] In the embodiment, three camera views are given to form observation images;

[0051] Step 2: Construct a robot arm multi-view fusion model, introduce the original multi-view representation into the state representation and the position encoding to form a multi-view position state representation; pass the multi-view position state representation through a multi-head self-attention fusion layer to form a state embedding, and use a double analog metric and a mask and reconstruction loss to optimize the multi-view representation fusion;

[0052] Step 2.1: Construct an original multi-view set as shown in formula (1) using the observation images;

[0053]

[0054] Where o i represents the observation image of the i-th view, H, W, and C represent the height, width, and channel number of the image, respectively;

[0055] In the embodiment, each multi-view is composed of an RGB observation image with a height of 128, a width of 128, and 3 channels;

[0056] Step 2.2: Construct a convolutional encoder and a multi-head self-attention fusion module, encode the original multi-view set using a convolutional neural network, map the original view to an embedding vector, and form an original multi-view representation as shown in formula (2);

[0057] x i ={x 1 ,x 2 ,...,x k} (2)

[0058] Where, d is the embedding dimension of the hidden space;

[0059] Step 2.3: Introduce the original multi-view representation into a state representation to be trained and a one-dimensional position encoding E pos to form a multi-view position state representation as shown in formula (3);

[0060]

[0061] Step 2.4: A multi-head self-attention fusion layer is constructed to pass the multi-view position state representation through the fusion layer as shown in equations (4) and (5) to form the state embedding;

[0062] z ′ l = MHSA(LN(z l-1 ))+z l-1 l = 1... L (4)

[0063] z l = MLP(LN(z l ))+z l· l = 1... L (5)

[0064] wherein MHSA(·) represents a multi-head self-attention layer; LN(·) represents normalization processing on the features; MLP(·) represents a multi-layer perception; L represents the number of multi-head self-attention layers; and is taken as the state embedding;

[0065] Step 2.5: The multi-view representation fusion is optimized using a dual analog metric;

[0066] Step 2.5.1: Two multi-views

[0067] Step 2.5.2: The similarity metric distance of the two multi-views is obtained in the manner as shown in equation (6); and the two multi-views produce the same probability distribution under the same action as the determination condition of the equivalent state representation;

[0068]

[0069] wherein φ ω (·) represents a convolutional encoder and a multi-head self-attention fusion module; and ω represents a model parameter;

[0070] Step 2.5.3: The φ ω (·) fusion module is optimized in the manner as shown in equation (7);

[0071]

[0072] wherein L represents a dual analog fusion loss; and γ is a weight coefficient; the dynamics model in the latent space, is a replay buffer; a represents a task action; s i′ s j′ represents the state representation of the latent space at the next moment;

[0073] Step 2.6: optimizing multi-view representation fusion with mask and reconstruction loss;

[0074] Step 2.6.1: randomly performing pixel masking on the original multi-view set to form a mask observation sequence

[0075] Step 2.6.2: passing the mask observation sequence through a convolutional encoder and a multi-head self-attention fusion module to form a masked state embedding

[0076] Step 2.6.3: constructing a prediction head model to obtain a reconstructed state embedding using the prediction head model

[0077] Step 2.6.4: optimizing the prediction head model and using a mask reconstruction loss as shown in equation (8) ω (·) the fusion module is optimized simultaneously;

[0078]

[0079] Step 2.7: establishing a joint optimization objective as shown in equation (9)

[0080]

[0081] where λ represents the weighting coefficient of the reconstruction loss;

[0082] In the embodiment, the three multi-view observation images are respectively mapped to embedding vectors with a dimension of 50 through a convolutional neural network, and the multi-view position state representation z0 is obtained through two layers of multi-head self-attention layers as the state embedding. The weight coefficient γ in the dual simulation metric optimization function is set to 0.99, and the weighting coefficient λ of the reconstruction loss in the joint optimization objective is 1;

[0083] Step 3: training the multi-view fusion model of the robot arm using the joint optimization objective;

[0084] Step 4: inputting the multi-view set into the trained multi-view fusion model of the robot arm to output a multi-view fusion representation; and using the multi-view fusion representation for robot arm task control;

[0085] In the embodiment, the multi-view fusion representation is input into a downstream reinforcement learning method (such as PPO or SAC), and after obtaining the current control action and interacting with the environment, the next optimization is performed.

[0086] To further demonstrate the superiority of the present application, the practicability of the present application is verified and explained by simulation.

[0087] For example, Figure 2As shown, the simulation environment adopts a set of 3D robotic arm operating environments Meta-World, and the final reward of each round is used as the evaluation index. The multi-view reinforcement learning methods compared by the present application include: 1) Keypoint3D method of learning 3D key points from multiple third-person perspective cameras; 2) LookCloser method of applying cross-attention mechanism between view pairs to integrate multi-view information; 3) Fuse2Control (F2C) method of learning state space model using information theory method; 4) MVD, a method for decoupling multi-view representation based on contrastive learning. In addition, we also compare common reinforcement learning algorithms by stacking all three views of images to form observations: 5) RAD, which achieves high sample efficiency through data augmentation; and 6) the original PPO algorithm.

[0088] The present application is always superior to other multi-view reinforcement learning algorithms in all five tasks, showing significantly higher sample efficiency and more stable performance. In particular, the ordinary PPO algorithm shows almost no signs of performance improvement, indicating that it is difficult to extract meaningful fusion representation without any special design. Although Random Data Augmentation (RAD) integrates data augmentation techniques and performs well in simpler tasks, it is difficult to learn effective fusion representation in more complex tasks such as "window opening" and "box closing". Keypoint3D performs competitively in some tasks, especially in the "box closing" task, but its overall training efficiency and final performance are still not ideal. LookCloser, also based on the Transformer architecture, is also effective. However, its overall performance has not yet exceeded the present application. In addition, the present application is superior to the previous optimal algorithm F2C in terms of convergence performance in most tasks.

[0089] The above results and analysis show that the multi-view representation fusion method applied to robotic arms proposed by the present application can effectively improve the efficiency of reinforcement learning in robotic arm task execution.

[0090] The above specific description further details the purpose, technical solutions and benefits of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A multi-view representation fusion method applied to a robotic arm, characterized by: The following steps are included: Step 1: Use the multi-view camera of the robotic arm to capture images in real time to form an observation image; Step 2: Build a multi-view fusion model for the robotic arm, introduce the original multi-view representation into the state representation and position encoding to form a multi-view position state representation; pass the multi-view position state representation through a multi-head self-attention fusion layer to form a state embedding, and use the dual analog metric and mask and reconstruction loss to optimize the multi-view representation fusion; Step 2.1: Use the observed images to construct the original multi-view set as shown in formula (1); Among them i represents the observed image of the i-th perspective, H, W, and C represent the height, width, and number of channels of the image respectively; Step 2.2: Construct a convolutional encoder and a multi-head self-attention fusion module, use a convolutional neural network to encode the original multi-view set, map the original view into an embedding vector, and form the original multi-view representation as shown in Equation (2); x i ={x 1 ,x 2 ,...,x k } (2) in, d is the embedding dimension of the latent space; Step 2.3: Introduce the original multi-view representation into a state representation to be trained and one-dimensional position encoding E pos , forming a multi-view position state representation as shown in formula (3); Step 2.4: Construct a multi-head self-attention fusion layer and pass the multi-view position state representation through the fusion layer as shown in Equations (4) and (5) to form a state embedding; Among them, MHSA(·) represents the multi-head self-attention layer; LN(·) represents the normalization of features; MLP(·) represents the multi-layer perceptron; L represents the number of multi-head self-attention layers; Embedded as state; Step 2.5: Optimize the multi-view representation fusion using dual-analog metrics; Step 2.6: Optimize the multi-view representation fusion using mask and reconstruction losses; Step 2.7: Establish the joint optimization objective as shown in formula (9); Among them, λ represents the weighted coefficient of reconstruction loss; Step 3: Use the joint optimization objective to train the multi-view fusion model of the robotic arm; Step 4: Input the multi-view set into the trained robotic arm multi-view fusion model and output the multi-view fusion representation; use the multi-view fusion representation for robotic arm task control.

2. The multi-view representation fusion method for a robotic arm according to claim 1, characterized in that: The implementation method of step 2.5 is as follows: Step 2.5.1: Randomly select two multi-views from the original multi-view set Step 2.5.2: Obtain two multi-views using the method shown in formula (6) Similarity metric distance of two multi-view The same probability distribution is generated under the same action as the judgment condition for equivalent state representation; Among them, φ ω (·) represents the convolutional encoder and multi-head self-attention fusion module; ω represents the model parameter; Step 2.5.3: Use the method shown in formula (7) to calculate φ ω (·) Fusion modules for optimization; in, represents the dual simulation fusion loss; γ weight coefficient; Dynamical models in latent space, is the playback buffer; a represents the task action; s i′ ,s j′ Represents the state representation of the latent space at the next moment.

3. The multi-view representation fusion method for a robotic arm according to claim 1, characterized in that: Step 2.6 is implemented as follows: Step 2.6.1: Randomly mask the original multi-view set to form a mask observation sequence Step 2.6.2: Pass the masked observation sequence through the convolutional encoder and the multi-head self-attention fusion module to form the masked state embedding Step 2.6.3: Build a prediction head model and use it to obtain the reconstructed state embedding Step 2.6.4: Use the mask reconstruction loss as shown in Equation (8) to predict the head model and φ ω (·) The fusion modules are optimized simultaneously.

Images