Training method and system of control model, electronic equipment and storage medium

Abstract

The invention provides a training method and system of a control model, electronic equipment and a storage medium. The training method of the control model comprises the steps that a first image shot by a first camera and a second image shot by a second camera in a first view field are collected, the first camera is arranged on a robot, and the second camera is away from the robot by a preset distance and is fixedly arranged; performing first training operation on the first control model according to the first image and the second image to obtain a second control model; acquiring a third image shot by the second camera in the second field of view; and performing second training operation on the second control model according to the first image and the third image to obtain a target control model. According to the invention, the first training operation is carried out on the control model by using the conventional image, and then the second training operation is carried out on the control model by using the image after the view field is changed, so that the target control model can be applied to a specific environment and can also adapt to the environment after the view field is changed, and the target control model has certain generalization ability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a control model training method and system, an electronic device and a storage medium. BACKGROUND

[0002] In recent years, vision-based end-to-end learning algorithms have made significant progress in robot manipulation, dexterous manipulation and mobility. However, these methods still face challenges in policy generalization. Policy generalization ability refers to the ability of a control model to effectively use existing knowledge to make decisions when encountering new environments or new tasks. In practical applications, robots may face different field of view challenges, which requires the control model not only to perform well in a specific training environment, but also to have a certain generalization ability. SUMMARY

[0003] The present application aims to at least solve one of the problems in the prior art or related art.

[0004] To this end, a first aspect of the present application provides a control model training method.

[0005] A second aspect of the present application provides a control model training system.

[0006] A third aspect of the present application provides an electronic device.

[0007] A fourth aspect of the present application provides a storage medium.

[0008] A fifth aspect of the present application provides a computer program product.

[0009] Therefore, according to a first aspect of the present application, a control model training method is provided, comprising: collecting a first image photographed by a first camera and a second image photographed by a second camera in a first field of view, wherein the first camera is arranged on a robot, and the second camera is arranged at a preset distance from the robot and is fixedly arranged; performing a first training operation on a first control model according to the first image and the second image to obtain a second control model; collecting a third image photographed by the second camera in a second field of view; and performing a second training operation on the second control model according to the first image and the third image to obtain a target control model.

[0010] The application provides a training method of a control model, which mainly comprises the following steps: first, collecting a first image and a second image, wherein the first image is obtained by a first camera arranged on a robot, and the first camera can be an eye-in-hand camera arranged at a terminal joint of the robot; since the first camera is arranged on the robot, when the robot moves, the shooting angle of the first camera also changes, so the first image can be a first-person perspective image; the second image is obtained by a second camera with a preset distance from the robot and fixedly arranged under a first field of view, that is, the second camera is a camera with a fixed pose, so the second image can be a third-person perspective image; the preset distance is the distance at which the second camera can shoot the whole robot; the field of view refers to the angle or area that can be covered by the camera lens at a specific position and direction; after obtaining the first image and the second image, a first training operation is performed on a first control model according to the first image and the second image to obtain a second control model; the first training operation is performed on the first control model by using the first image and the second image, so that the obtained second control model can obtain more comprehensive environmental information according to the first image and the second image, thereby enabling the second control model to perform better in a specific environment; then, a third image shot by the second camera under a second field of view is collected, wherein the second field of view is different from the first field of view; then, a second training operation is performed on the second control model according to the first image shot by the first camera and the third image shot by the second camera to obtain a target control model; the second training operation is performed on the second control model by using the first image and the third image with a changed perspective, so that the target control model can adapt to environments with different perspectives. The first training operation is performed on the first control model by using the first image and the second image to obtain the second control model, and then the second training operation is performed on the second control model by using the third image with a changed perspective and the first image to obtain the target control model, so that the target control model can be applied to a specific environment and adapt to an environment with a changed field of view, thereby enabling the target control model to have a certain generalization ability.

[0011] According to the training method of the control model, the following technical features can be further provided.

[0012] In some technical solutions, optionally, the step of performing a first training operation on a first control model based on a first image and a second image to obtain a second control model includes: initializing the first control model, wherein the first control model includes at least one of a fusion feature network, an action generation network, an evaluation network, a target evaluation network, or an inverse dynamics network; determining a fusion feature set based on the first image, the second image, and the first control model; determining a first control signal based on the fusion feature set and the first control model; controlling the robot to execute the first control signal and calculating a feedback signal based on the executed robot; updating the first control model based on the fusion feature set, the first control signal, and the feedback signal to obtain a third control model; re-acquiring the first image and the second image and inputting them into the third control model for retraining until the training count reaches a preset number of training times, and then using the third control model as the second control model. In this technical solution, the step of performing a first training operation on a first control model based on a first image and a second image to obtain a second control model includes: First, initializing the parameters in the first control model, wherein the first control model includes at least one combination of a fusion feature network, an action generation network, an evaluation network, a target evaluation network, or an inverse dynamics network; the fusion feature network can obtain the robot's fusion features based on the first and second images; the action generation (Actor) network can output action values ​​based on the current state, i.e., it is responsible for generating the robot's control signals; the evaluation (Critic) network can evaluate the actions selected by the Actor network, i.e., it predicts the future cumulative reward based on the current state and actions; the target evaluation (Critic) network can provide stable target values; and the inverse dynamics network can perform precise control. After initializing the first control model, the first and second images are input into the first control model. In the first control model, the image features of the first image and the image features of the second image are fused to obtain a fusion feature set, which can then be used to obtain more comprehensive environmental information. Then, a first control signal is obtained based on the fusion feature set and the first control model, and the robot's control action is obtained in the first control model based on the fusion feature set, i.e., the robot's environmental information. Then, the robot executes the corresponding control action according to the first control signal, and calculates the feedback signal based on the robot's current state and the object's position information after the control action is completed. The first control model is then updated based on the fused feature set, the first control signal, and the feedback signal to obtain the third control model. This completes one training cycle. Finally, the first and second images are re-acquired, and then the re-acquired first and second images are input into the third control model, repeating the above steps until the preset number of training iterations is reached (the preset number of training iterations can be 1 million). The final third control model is then used as the second control model.The second control model is obtained by training the first control model, which enables the second control model to perform well in the characteristic environment.

[0013] In some technical solutions, optionally, the step of determining the fusion feature set based on the first image, the second image, and the first control model includes: determining a first feature vector based on the first image; determining a second feature vector based on the second image; determining a first attention feature and a second attention feature based on the first feature vector, the second feature vector, and the fusion feature network, wherein the fusion feature network includes two convolutional neural networks and two sub-attention networks; and determining the fusion feature set based on the first attention feature and the second attention feature.

[0014] In this technical solution, the step of determining the fused feature set based on the first image, the second image, and the first control model includes: firstly, determining the first feature vector of the first image based on the first image, and determining the second feature vector of the second image based on the second image. The fused feature network includes two CNN (Convolutional Neural Networks) encoding networks, two subsequent fully connected networks, and two sub-attention networks. Firstly, the first image and the second image are respectively input into the two CNN encoding networks to obtain the first feature vector of the first image and the second feature vector of the second image. Then, the first feature vector is used as a query, and the second feature vector is used as a key-value pair (key and value) input into one sub-attention network. This input is then processed through two subsequent fully connected networks to obtain the first attention feature. The second feature vector is then used as a query, and the first feature vector is used as a key-value pair (key and value) input into another sub-attention network. This input is then processed through two subsequent fully connected networks to obtain the second attention feature. Finally, the first attention feature and the second attention feature are added together to obtain the fused feature set. The present invention employs a visual information fusion method with a cross-view attention mechanism, which enables the extraction of image features obtained from a third-person perspective to process and pay attention to information from a first-person perspective, thereby allowing information from two different perspectives to interact and integrate, and jointly influence the downstream robot's operation strategy.

[0015] In some technical solutions, optionally, the step of determining the first control signal based on the fusion feature set and the first control model includes: determining the first fusion feature of the current frame based on the fusion feature set; determining the mean vector and variance vector of the robot's control strategy based on the first fusion feature and the action generation network; establishing a multidimensional Gaussian distribution of the control strategy based on the mean vector and variance vector; and determining the first control signal based on the multidimensional Gaussian distribution.

[0016] In this technical solution, the step of determining the first control signal based on the fused feature set and the first control model includes: firstly, obtaining the first fused feature of the current frame based on the fused feature set; then, inputting the first fused feature into the action generation network to obtain the mean vector and variance vector of the robot control strategy; then, constructing a multidimensional Gaussian distribution of the control strategy using the mean vector and variance vector; and finally, obtaining the first control signal by sampling from the multidimensional Gaussian distribution. By using the action generation network to determine the first control signal, the accuracy of the first control signal is improved.

[0017] In some technical solutions, optionally, the step of updating the first control model based on the fusion feature set, the first control signal, and the feedback signal to obtain the third control model includes: updating the fusion feature network and the inverse dynamics network based on the fusion feature set and the first control signal; updating the evaluation network and the fusion feature network based on the fusion feature set and the feedback signal; updating the action generation network based on the updated evaluation network and the fusion feature set; and obtaining the third control model based on the target evaluation network, the updated fusion feature network, the updated inverse dynamics network, the updated evaluation network, and the updated action generation network.

[0018] In this technical solution, the step of updating the first control model based on the fusion feature set, the first control signal, and the feedback signal to obtain the third control model includes: updating the parameters in the fusion feature network and the inverse dynamics network using the fusion feature set and the first control signal, so that the outputs of the fusion feature network and the inverse dynamics network are closer to the expectation. Then, the evaluation network and the target evaluation network are updated using the fusion feature set and the calculated feedback signal, and the fusion feature network is updated a second time. Simultaneously, the action generation network is updated using the updated evaluation network and the fusion feature set. Finally, the updated multiple networks are combined, i.e., the third control model is generated based on the target evaluation network, the updated fusion feature network, the updated inverse dynamics network, the updated evaluation network, and the updated action generation network. By updating the first control model to obtain the third control model, the third control model can approach a control model that meets the requirements.

[0019] In some technical solutions, optionally, the step of updating the fusion feature network and the inverse dynamics network according to the fusion feature set and the first control signal includes: determining the second fusion feature of the next frame according to the fusion feature set; determining the predicted second control signal according to the first fusion feature, the second fusion feature and the inverse dynamics network; calculating the first error between the second control signal and the first control signal; and updating the fusion feature network and the inverse dynamics network respectively according to the first error.

[0020] In this technical solution, the steps of updating the fusion feature network and the inverse dynamics network based on the fusion feature set and the first control signal include: firstly, determining the second fusion feature for the next frame from the fusion feature set; then, concatenating the first and second fusion features and placing them into the inverse dynamics network to predict the current robot action, i.e., the second control signal; then, using the first control signal, i.e., the robot's actual executed action, as the target value; and finally, calculating the first error between the second control signal and the target value, i.e., the prediction error of the inverse dynamics network. Finally, updating the parameters in the inverse dynamics network and the fusion feature network using gradient descent based on the prediction error of the inverse dynamics network yields the updated inverse dynamics network and the updated fusion feature network. The update of the inverse dynamics network is achieved by calculating the difference between the second control signal predicted by the inverse dynamics network and the first control signal.

[0021] In some technical solutions, optionally, the step of updating the evaluation network and the fusion feature network based on the fusion feature set and the feedback signal includes: obtaining a target value estimation function based on the second fusion feature, the feedback signal and the target evaluation network; obtaining a first loss function based on the first fusion feature, the evaluation network and the target value estimation function; and updating the fusion feature network and the evaluation network respectively based on the first loss function.

[0022] In this technical solution, the steps of updating the evaluation network and the fusion feature network based on the fusion feature set and the feedback signal include: firstly, inputting the second fusion feature into the target evaluation network; then, combining the feedback signal obtained after the robot performs the corresponding action according to the first control signal to obtain the target value estimation function; then, inputting the first fusion feature and the target value estimation function into the evaluation network, calculating the value function estimate and loss, i.e., the first loss function; and finally, updating the fusion feature network, the evaluation network, and the target evaluation network based on the first loss function.

[0023] In some technical solutions, optionally, the step of updating the action generation network based on the updated evaluation network and the fused feature set includes: determining the robot's control policy distribution based on the first fused features and the action generation network; determining an estimate of the current control policy based on the updated evaluation network and the control policy distribution; determining a second loss function based on the estimate; and updating the action generation network based on the second loss function.

[0024] In this technical solution, the steps of updating the action generation network based on the updated evaluation network and the fused feature set include: firstly, inputting the first fused feature into the action generation network to calculate the robot control policy distribution; then inputting the calculated control policy distribution into the updated evaluation network to obtain an estimate of the current control policy; then calculating the loss of the action generation network, i.e., the second loss function, based on the estimate; and finally updating the action generation network using gradient descent based on the second loss function to obtain the updated action generation network.

[0025] In some technical solutions, optionally, the step of performing a second training operation on the second control model based on the first image and the third image to obtain the target control model includes: freezing the action generation network and the inverse dynamics network in the second control model; performing a first training operation on the frozen second control model based on the first image and the third image, and updating the fusion feature network in the second control model to obtain a fourth control model; re-acquiring the first image and the third image, and inputting them into the fourth control model for retraining until the training number reaches a preset number of training times, and then using the fourth control model as the target control model.

[0026] In this technical solution, the step of performing a second training operation on the second control model based on the first and third images to obtain the target control model includes: firstly, freezing the action generation network and inverse dynamics network in the second control model, i.e., keeping the parameters in the action generation network and inverse dynamics network of the second control model unchanged. Then, re-performing the first training operation on the second control model using the first and third images, i.e., re-executing the first training operation steps using the first and third images. Since the action generation network and inverse dynamics network are frozen, only the fusion feature network in the second control model is updated during training. Then, a fourth control model is obtained based on the updated fusion feature network, the frozen action generation network, and the frozen inverse dynamics network. Finally, the first and third images are re-acquired and input into the fourth control model for retraining until the preset number of training iterations is reached (e.g., 1 million iterations), and the fourth control model is used as the target control model. By training the second control model using the third and first images after viewpoint changes, and by updating only the fusion feature network during training, the target control model can perform well in environments with different viewpoints.

[0027] According to a second aspect of the present invention, a training system for a control model is proposed, wherein the training system for the control model includes: a first acquisition module, which is used to acquire a first image captured by a first camera and a second image captured by a second camera in a first field of view, wherein the first camera is mounted on a robot and the second camera is fixed at a preset distance from the robot; a first training module, which is used to perform a first training operation on the first control model based on the first image and the second image to obtain a second control model; a second acquisition module, which is used to acquire a third image captured by the second camera in a second field of view; and a second training module, which is used to perform a second training operation on the second control model based on the first image and the third image to obtain a target control model.

[0028] The control model training system provided by this invention mainly includes: a first acquisition module, a first training module, a second acquisition module, and a second training module. The first acquisition module can acquire a first image and a second image. The first image is captured by a first camera mounted on the robot. The first camera can be a hand-eye camera mounted at the robot's end joint. Since the first camera is mounted on the robot, its shooting angle changes as the robot moves, thus the first image is a first-person perspective image. The second image is captured by a second camera fixed at a preset distance from the robot, within a first field of view. That is, the second camera is a fixed-position camera, thus the second image is a third-person perspective image. The preset distance is the distance at which the second camera can capture the entire robot. The field of view refers to the angle or area that a camera lens can cover at a specific position and direction. After obtaining the first and second images, the first training module performs a first training operation on the first control model based on the first and second images to obtain a second control model. By using the first and second images to perform the first training operation on the first control model, the obtained second control model can obtain more comprehensive environmental information based on the first and second images, thereby enabling the second control model to perform better in specific environments. Then, the second acquisition module acquires a third image captured by the second camera in a second field of view, which is different from the first field of view. The second training module then performs a second training operation on the second control model based on the first image captured by the first camera and the third image captured by the second camera, obtaining the target control model. By retraining the second control model using the first image and the third image with a changed viewpoint, the target control model can adapt to environments with different viewpoints. This invention first uses the first and second images to perform a first training operation on the first control model to obtain the second control model, and then uses the third image with a changed viewpoint and the first image to perform a second training operation on the second control model to obtain the target control model. This allows the target control model to be applied to both specific environments and environments with changed viewpoints, thus giving the target control model a certain degree of generalization ability.

[0029] According to a third aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a training method for a control model as described above.

[0030] The electronic device provided by this invention, in which the processor executes a computer program to implement the steps of the above-mentioned control model training method, can achieve the technical effects of any of the above technical solutions, and will not be elaborated further here.

[0031] According to a fourth aspect of the invention, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the steps of the training method for the control model as described above.

[0032] The storage medium provided by this invention, when the computer program is executed by the processor, implements the steps of the above-described control model training method, and can achieve the technical effects of any of the above technical solutions, which will not be elaborated here.

[0033] According to a fifth aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of a training method for a control model as described in any of the above-described technical solutions.

[0034] The computer program product provided by this technical solution implements the steps of the training method of the control model of any technical solution of the present invention, and therefore has all the beneficial effects of the training method of the control model of any technical solution of the present invention, which will not be repeated here.

[0035] Additional aspects and advantages of the invention will become apparent in the following description or may be learned by practice of the invention. Attached Figure Description

[0036] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0037] Figure 1 A flowchart illustrating a training method for a control model according to an embodiment of the present invention is shown.

[0038] Figure 2 The diagram illustrates a step in a training method for a control model according to an embodiment of the present invention: performing a first training operation on a first control model based on a first image and a second image to obtain a second control model.

[0039] Figure 3 The diagram illustrates a step in a training method for a control model according to an embodiment of the present invention, which involves determining a fusion feature set based on a first image, a second image, and a first control model.

[0040] Figure 4 The diagram illustrates a step in a training method for a control model according to an embodiment of the present invention, which involves determining a first control signal based on a fused feature set and a first control model.

[0041] Figure 5 The diagram illustrates a step in a training method for a control model according to an embodiment of the present invention: updating the first control model based on a fused feature set, a first control signal, and a feedback signal to obtain a third control model.

[0042] Figure 6 The diagram illustrates a step in the training method of a control model according to an embodiment of the present invention: updating the fusion feature network and the inverse dynamics network based on the fusion feature set and the first control signal.

[0043] Figure 7 The diagram illustrates a step in the training method of a control model according to an embodiment of the present invention: updating the evaluation network and the fusion feature network based on the fusion feature set and the feedback signal.

[0044] Figure 8 The diagram illustrates a step in the training method of a control model according to an embodiment of the present invention: updating the action generation network based on the updated evaluation network and the fused feature set.

[0045] Figure 9 The diagram illustrates a step in a training method for a control model according to an embodiment of the present invention: performing a second training operation on a second control model based on a first image and a third image to obtain a target control model.

[0046] Figure 10 A schematic block diagram of a training system for a control model according to an embodiment of the present invention is shown;

[0047] Figure 11 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0048] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0049] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0050] Figure 1 A flowchart illustrating a training method for a control model according to an embodiment of the present invention is shown. The method includes:

[0051] Step 102: Acquire a first image captured by the first camera and a second image captured by the second camera in the first field of view, wherein the first camera is mounted on the robot and the second camera is fixed at a preset distance from the robot;

[0052] Step 104: Perform a first training operation on the first control model based on the first image and the second image to obtain the second control model;

[0053] Step 106: Acquire the third image captured by the second camera in the second field of view;

[0054] Step 108: Perform a second training operation on the second control model based on the first and third images to obtain the target control model.

[0055] The training method for the control model provided by this invention mainly includes: First, acquiring a first image and a second image. The first image is captured by a first camera mounted on the robot. The first camera can be a hand-eye camera mounted at the robot's end joint. Since the first camera is mounted on the robot, its shooting angle changes as the robot moves, thus the first image can be a first-person perspective image. The second image is captured by a second camera fixed at a preset distance from the robot, within a first field of view. That is, the second camera is a fixed-position camera, thus the second image can be a third-person perspective image. The preset distance is the distance at which the second camera can capture the entire robot. The field of view refers to the angle or area that a camera lens can cover in a specific position and direction. After obtaining the first and second images, a first training operation is performed on the first control model based on the first and second images to obtain a second control model. By using the first and second images to perform the first training operation on the first control model, the obtained second control model can obtain more comprehensive environmental information based on the first and second images, thereby enabling the second control model to perform better in specific environments. Then, a third image is acquired by the second camera in a second field of view, where the second field of view is different from the first field of view. Then, based on the first image captured by the first camera and the third image captured by the second camera, a second training operation is performed on the second control model to obtain the target control model. By using the first image and the third image after the viewpoint change, the second control model is trained again, enabling the target control model to adapt to environments with different viewpoints. This invention first uses the first and second images to perform a first training operation on the first control model to obtain the second control model, and then uses the third image after the viewpoint change and the first image to perform a second training operation on the second control model to obtain the target control model. This allows the target control model to be applied to both specific environments and environments with changed viewpoints, thus giving the target control model a certain degree of generalization ability.

[0056] Figure 2The diagram illustrates a flowchart of a control model training method according to an embodiment of the present invention, showing the step of performing a first training operation on a first control model based on a first image and a second image to obtain a second control model; wherein, the step of performing the first training operation on the first control model based on the first image and the second image to obtain the second control model includes:

[0057] Step 202: Initialize the first control model, wherein the first control model includes at least one of the following: a fusion feature network, an action generation network, an evaluation network, a target evaluation network, or an inverse dynamics network;

[0058] Step 204: Determine the fusion feature set based on the first image, the second image, and the first control model;

[0059] Step 206: Determine the first control signal based on the fused feature set and the first control model;

[0060] Step 208: Control the robot to execute the first control signal, and calculate the feedback signal based on the robot after execution;

[0061] Step 210: Update the first control model based on the fused feature set, the first control signal, and the feedback signal to obtain the third control model;

[0062] Step 212: Reacquire the first and second images and input them into the third control model for retraining until the preset number of training iterations is reached. Then, use the third control model as the second control model.

[0063] In this embodiment, the step of performing a first training operation on the first control model based on the first image and the second image to obtain the second control model includes: First, initializing the parameters in the first control model, wherein the first control model includes at least one combination of a fusion feature network, an action generation network, an evaluation network, a target evaluation network, or an inverse dynamics network; the fusion feature network can obtain the robot's fusion features based on the first image and the second image; the action generation (Actor) network can output action values ​​based on the current state, i.e., it is responsible for generating the robot's control signals; the evaluation (Critic) network can evaluate the actions selected by the Actor network, i.e., it predicts the future cumulative reward based on the current state and actions; the target evaluation (Critic) network can provide stable target values; and the inverse dynamics network can perform precise control. After initializing the first control model, the first image and the second image are input into the first control model. In the first control model, the image features of the first image and the image features of the second image are fused to obtain a fusion feature set, which can then be used to obtain more comprehensive environmental information. Then, a first control signal is obtained based on the fusion feature set and the first control model, and the robot's control action is obtained in the first control model based on the fusion feature set, i.e., the robot's environmental information. Then, the robot executes corresponding control actions based on the first control signal, and calculates feedback signals based on the robot's current state and the object's position information after the control actions are completed. The first control model is then updated based on the fused feature set, the first control signal, and the feedback signal to obtain the third control model. This completes one training iteration. Finally, the first and second images are re-acquired, and then input into the third control model, repeating the above steps until the preset number of training iterations is reached (e.g., 1 million). The final third control model is then used as the second control model. By training the first control model to obtain the second control model, the second control model can exhibit good performance in the featured environment.

[0064] Figure 3 The diagram illustrates a step in a control model training method according to an embodiment of the present invention: determining a fusion feature set based on a first image, a second image, and a first control model. The step of determining the fusion feature set based on the first image, the second image, and the first control model includes:

[0065] Step 302: Determine the first feature vector based on the first image;

[0066] Step 304: Determine the second feature vector based on the second image;

[0067] Step 306: Determine the first attention feature and the second attention feature based on the first feature vector, the second feature vector, and the fused feature network, wherein the fused feature network includes two convolutional neural networks and two sub-attention networks;

[0068] Step 308: Determine the fusion feature set based on the first attention feature and the second attention feature.

[0069] In this embodiment, the step of determining the fused feature set based on the first image, the second image, and the first control model includes: firstly, determining the first feature vector of the first image based on the first image, and determining the second feature vector of the second image based on the second image. The fused feature network includes two CNN (Convolutional Neural Networks) encoding networks, two subsequent fully connected networks, and two sub-attention networks. Firstly, the first image and the second image are respectively input into the two CNN encoding networks to obtain the first feature vector of the first image and the second feature vector of the second image. Then, the first feature vector is used as a query, and the second feature vector is used as a key-value pair (key and value) input into one sub-attention network. This input is then processed through two subsequent fully connected networks to obtain the first attention feature. The second feature vector is then used as a query, and the first feature vector is used as a key-value pair (key and value) input into another sub-attention network. This input is then processed through two subsequent fully connected networks to obtain the second attention feature. Finally, the first attention feature and the second attention feature are added together to obtain the fused feature set. The present invention employs a visual information fusion method with a cross-view attention mechanism, which enables the extraction of image features obtained from a third-person perspective to process and pay attention to information from a first-person perspective, thereby allowing information from two different perspectives to interact and integrate, and jointly influence the downstream robot's operation strategy.

[0070] Figure 4 The diagram illustrates a step in a control model training method according to an embodiment of the present invention, specifically the step of determining a first control signal based on a fused feature set and a first control model; wherein the step of determining the first control signal based on the fused feature set and the first control model includes:

[0071] Step 402: Determine the first fusion feature of the current frame based on the fusion feature set;

[0072] Step 404: Determine the mean vector and variance vector of the robot's control strategy based on the first fusion feature and the action generation network;

[0073] Step 406: Establish a multidimensional Gaussian distribution of the control strategy based on the mean vector and variance vector;

[0074] Step 408: Determine the first control signal based on the multidimensional Gaussian distribution.

[0075] In this embodiment, the step of determining the first control signal based on the fused feature set and the first control model includes: firstly, obtaining the first fused feature of the current frame based on the fused feature set; then, inputting the first fused feature into the action generation network to obtain the mean vector and variance vector of the robot control strategy; then, constructing a multidimensional Gaussian distribution of the control strategy using the mean vector and variance vector; and finally, obtaining the first control signal by sampling from the multidimensional Gaussian distribution. By using the action generation network to determine the first control signal, the accuracy of the first control signal is improved.

[0076] Figure 5 The diagram illustrates a flowchart of a training method for a control model according to an embodiment of the present invention, showing the step of updating a first control model based on a fused feature set, a first control signal, and a feedback signal to obtain a third control model. The step of updating the first control model based on the fused feature set, the first control signal, and the feedback signal to obtain the third control model includes:

[0077] Step 502: Update the fusion feature network and the inverse dynamics network according to the fusion feature set and the first control signal;

[0078] Step 504: Update the evaluation network and the fusion feature network based on the fusion feature set and feedback signals;

[0079] Step 506: Update the action generation network based on the updated evaluation network and the fused feature set;

[0080] Step 508: Obtain the third control model based on the target evaluation network, the updated fusion feature network, the updated inverse dynamics network, the updated evaluation network, and the updated action generation network.

[0081] In this embodiment, the step of updating the first control model based on the fusion feature set, the first control signal, and the feedback signal to obtain the third control model includes: updating the parameters in the fusion feature network and the inverse dynamics network using the fusion feature set and the first control signal, so that the outputs of the fusion feature network and the inverse dynamics network are closer to the expectation. Then, the evaluation network and the target evaluation network are updated using the fusion feature set and the calculated feedback signal, and the fusion feature network is updated a second time. At the same time, the action generation network is updated using the updated evaluation network and the fusion feature set. Finally, the updated multiple networks are combined, that is, the third control model is generated based on the target evaluation network, the updated fusion feature network, the updated inverse dynamics network, the updated evaluation network, and the updated action generation network. By updating the first control model to obtain the third control model, the third control model can approach the control model that meets the requirements.

[0082] Figure 6 The diagram illustrates a step in a training method for a control model according to an embodiment of the present invention: updating the fused feature network and the inverse dynamics network based on a fused feature set and a first control signal. The step of updating the fused feature network and the inverse dynamics network based on the fused feature set and the first control signal includes:

[0083] Step 602: Determine the second fusion feature of the next frame based on the fusion feature set;

[0084] Step 604: Determine the predicted second control signal based on the first fusion feature, the second fusion feature, and the inverse dynamics network;

[0085] Step 606: Calculate the first error between the second control signal and the first control signal;

[0086] Step 608: Update the fusion feature network and the inverse dynamics network respectively based on the first error.

[0087] In this embodiment, the steps of updating the fusion feature network and the inverse dynamics network based on the fusion feature set and the first control signal include: firstly, determining the second fusion feature for the next frame from the fusion feature set; then, concatenating the first and second fusion features and placing them into the inverse dynamics network to predict the current robot action, i.e., the second control signal; then, using the first control signal, i.e., the robot's actual executed action, as the target value; and finally, calculating the first error between the second control signal and the target value, i.e., the prediction error of the inverse dynamics network. Finally, updating the parameters in the inverse dynamics network and the fusion feature network using gradient descent based on the prediction error of the inverse dynamics network yields the updated inverse dynamics network and the updated fusion feature network. The update of the inverse dynamics network is achieved by calculating the difference between the second control signal predicted by the inverse dynamics network and the first control signal.

[0088] Figure 7 The diagram illustrates a step in a training method for a control model according to an embodiment of the present invention: updating the evaluation network and the fusion feature network based on a fusion feature set and feedback signals. The step of updating the evaluation network and the fusion feature network based on the fusion feature set and feedback signals includes:

[0089] Step 702: Obtain the target value estimation function based on the second fusion feature, feedback signal, and target evaluation network;

[0090] Step 704: Obtain the first loss function based on the first fusion feature, the evaluation network, and the target value estimation function;

[0091] Step 706: Update the fusion feature network and the evaluation network according to the first loss function.

[0092] In this embodiment, the steps of updating the evaluation network and the fusion feature network based on the fusion feature set and the feedback signal include: firstly, inputting the second fusion feature into the target evaluation network; then, combining the feedback signal obtained after the robot performs the corresponding action according to the first control signal to obtain the target value estimation function; then, inputting the first fusion feature and the target value estimation function into the evaluation network, calculating the value function estimate and the loss, i.e., the first loss function; and finally, updating the fusion feature network, the evaluation network, and the target evaluation network based on the first loss function.

[0093] Figure 8 The diagram illustrates a step in a training method for a control model according to an embodiment of the present invention: updating the action generation network based on the updated evaluation network and the fused feature set. The step of updating the action generation network based on the updated evaluation network and the fused feature set includes:

[0094] Step 802: Determine the robot's control strategy distribution based on the first fusion feature and the action generation network;

[0095] Step 804: Determine the estimated value of the current control policy based on the updated evaluation network and control policy distribution;

[0096] Step 806: Determine the second loss function based on the estimated value;

[0097] Step 808: Update the action generation network according to the second loss function.

[0098] In this embodiment, the step of updating the action generation network based on the updated evaluation network and the fused feature set includes: firstly, inputting the first fused feature into the action generation network to calculate the robot control policy distribution; then, inputting the calculated control policy distribution into the updated evaluation network to obtain an estimate of the current control policy; then, calculating the loss of the action generation network, i.e., the second loss function, based on the estimate; and finally, updating the action generation network using gradient descent based on the second loss function to obtain the updated action generation network.

[0099] Figure 9 The diagram illustrates a flowchart of a control model training method according to an embodiment of the present invention, showing the step of performing a second training operation on a second control model based on a first image and a third image to obtain a target control model; wherein, the step of performing a second training operation on the second control model based on the first image and the third image to obtain the target control model includes:

[0100] Step 902: Freeze the action generation network and inverse dynamics network in the second control model;

[0101] Step 904: Perform the first training operation on the frozen second control model based on the first image and the third image, and update the fusion feature network in the second control model to obtain the fourth control model;

[0102] Step 906: Reacquire the first and third images and input them into the fourth control model for retraining until the preset number of training iterations is reached. Then, use the fourth control model as the target control model.

[0103] In this embodiment, the step of performing a second training operation on the second control model based on the first and third images to obtain the target control model includes: firstly, freezing the action generation network and inverse dynamics network in the second control model, i.e., keeping the parameters in the action generation network and inverse dynamics network of the second control model unchanged. Then, re-performing the first training operation on the second control model using the first and third images, i.e., re-executing the first training operation using the first and third images. Since the action generation network and inverse dynamics network are frozen, only the fusion feature network in the second control model is updated during training. Then, a fourth control model is obtained based on the updated fusion feature network, the frozen action generation network, and the frozen inverse dynamics network. Finally, the first and third images are re-acquired and input into the fourth control model for retraining until the training iterations reach a preset number of iterations, which can be 1 million. The fourth control model is then used as the target control model. By training the second control model using the third and first images after viewpoint changes, and by updating only the fusion feature network during training, the target control model is ensured to perform well in environments with different viewpoints.

[0104] For example, the training method provided by this invention is mainly applicable to desktop operation tasks of robotic arms equipped with two cameras. Taking the Reach task as an example, in the first training operation, the field of view of the third-person perspective camera, i.e., the second camera, is 50. The first training operation process is as follows: First, initialize the control model parameters. The control model includes a fusion feature network, which includes two CNN encoding networks f. θ1 f θ2 Two sub-attention networks ψ θ1 , ψ θ2 The control model also includes: Actor network π α Critic Network Q ω Target Critic Network and inverse dynamics model network D β Then, the feature network is fused to obtain the feature vectors of the first-view image (i.e., the first image) and the third-view image (i.e., the second image); the feature network also obtains the first-view image (i.e., the first image O1) from the camera at the end of the robotic arm (i.e., the first camera); the feature network then inputs the first-view image into the CNN encoder to obtain the feature vector f of the first-view image. θ1 (O1); The fusion feature network acquires the third-view image, i.e., the second image, from the fixed-pose camera, i.e., the second camera. O2; The fusion feature network inputs the third-view image into the CNN encoder to obtain the feature vector f of the third-view image. θ2(O2); then the feature fusion network calculates the fused features, and the specific steps are as follows: First, obtain the first-person view image feature vector as the query. The feature vector of a third-person perspective image is the key value ( and Then according to the formula The calculated attention feature T θ1 Where σ is the sigmoid function and norm represents the normalization function; then, the feature vector of the third-person view image is obtained as the query. The feature vector of a first-person perspective image is the key value ( and According to the formula The calculated attention feature T θ2 Where σ is the sigmoid function and norm represents the normalization function; finally, the two features are added together to obtain the fused feature T. θ =T θ1 +T θ2 After obtaining the fusion features, the fusion feature network extracts the fusion features of the current frame from the fusion features. The fused features are input into the Actor network to obtain the mean vector and variance vector of the robotic arm's control strategy; and a multidimensional Gaussian distribution π is constructed. α (T θ Finally, the control motion a of the robotic arm is obtained by sampling from it. t ~π α (T θ The robotic arm receives control signals from the server, namely the first control signal a. t The robotic arm executes the control signal and calculates the reward, i.e., the first feedback signal r, based on the object's position and the position of the robotic arm's gripper. t The feature fusion network acquires the next frame of first-person view image from the camera at the end of the robotic arm. The feature fusion network acquires the next frame of third-person view image from a camera with a fixed pose. The fusion feature network extracts the fusion features for the next frame from the fusion features.

[0105] Then the inverse dynamics network D β and the network T with fused feature networks θ To update, the specific steps are as follows: Combine the fused features of the current frame. fusion features with the next frame Connect the components and feed them into the inverse dynamics network to predict the robot arm's movements in the current frame. The target value is determined by the actual actions performed by the robotic arm, using the formula... The prediction error of the inverse dynamics network is calculated, where ||·|| is the L2 norm. Finally, the parameters β of the inverse dynamics network and the network parameters θ of the fused feature network are updated using gradient descent.

[0106] Simultaneously, the Critic network Q... ω and the network T with fused feature networks θ The update process involves the following steps: First, starting with the fused features from the next frame... Input the target Critic network, combined with the previously obtained reward r t Using formula Calculate the target value estimate; then fuse the features. Input into the Critic network and use the Bellman formula. The value function estimate and loss are calculated, and finally, gradient descent is used to update the network parameters θ of the fused feature network and the parameters ω of the Critic network. Simultaneously, the target Critic network is updated. Where τ = 0.995.

[0107] Then update the Actor network, with the following steps: fuse features. Input the policy distribution into the Actor network. The Critic network, combined with the policy distribution, calculates an estimate of the current policy. The Actor network loss is calculated based on the estimated values ​​as follows: in, This represents the loss value. Finally, gradient descent is used to update the Actor network parameters α.

[0108] Finally, the feature vectors of the first-person and third-person perspective images are reacquired, and environmental sampling and network updates are performed alternately until a fixed number of training steps of 1 million steps is reached. The inverse dynamics network parameters β, the Actor network parameters α, and the fusion feature network parameters θ are saved.

[0109] After the first training operation is completed, the field of view of the third-person camera is changed to 60, 70, 80, and 90 for the second training operation. The specific process is as follows: First, load network T from the trained fusion feature network. θ Actor Network π α and inverse dynamics network D β Then freeze the parameters of the Actor network and the inverse dynamics network, and update the network parameters θ of the fusion feature network through the first training operation; finally, repeat the above steps until a fixed number of adaptation steps of 100,000 steps is reached, and save the network parameters θ of the fusion feature network.

[0110] As shown in Table 1, Table 1 presents the task completion success rates of the first training operation, the second training operation, and the comparison method of the present invention in the Reach task. The comparison method is a multimodal reinforcement learning training method. As can be seen from Table 1, the task completion success rates of the first and second training operations provided by the present invention are significantly higher than those of the comparison method.

[0111] Table 1

[0112] Field of view values 50 60 70 80 90 Test average values Contrast method 0.771 0.739 0.599 0.042 0.024 0.351 First training operation 0.808 0.779 0.613 0.381 0.235 0.502 Second training operation 0.801 0.785 0.652 0.457 0.332 0.557

[0113] Figure 10 A schematic block diagram of a control model training system according to an embodiment of the present invention is shown; wherein, the control model training system 100 includes:

[0114] The first acquisition module 1002 is used to acquire a first image captured by a first camera and a second image captured by a second camera in a first field of view. The first camera is mounted on the robot, and the second camera is fixed at a preset distance from the robot.

[0115] The first training module 1004 is used to perform a first training operation on the first control model based on the first image and the second image to obtain the second control model.

[0116] The second acquisition module 1006 is used to acquire the third image captured by the second camera in the second field of view;

[0117] The second training module 1008 is used to perform a second training operation on the second control model based on the first image and the third image to obtain the target control model.

[0118] The control model training system 100 provided by this invention mainly includes: a first acquisition module 1002, a first training module 1004, a second acquisition module 1006, and a second training module 1008. The first acquisition module 1002 can acquire a first image and a second image. The first image is captured by a first camera mounted on the robot. The first camera can be a hand-eye camera mounted at the robot's end joint. Since the first camera is mounted on the robot, its shooting angle changes as the robot moves, thus the first image can be a first-person perspective image. The second image is captured by a second camera fixed at a preset distance from the robot, within a first field of view. That is, the second camera is a fixed-position camera, therefore the second image can be a third-person perspective image. The preset distance is the distance at which the second camera can capture the entire robot. Field of view refers to the angle or area that a camera lens can cover in a specific position and direction. After obtaining the first and second images, the first training module 1004 performs a first training operation on the first control model based on the first and second images to obtain a second control model. By using the first and second images for the first training operation, the obtained second control model can obtain more comprehensive environmental information based on the first and second images, thus enabling the second control model to perform better in specific environments. Then, the second acquisition module 1006 acquires a third image taken by the second camera in a second field of view, where the second field of view is different from the first field of view. Then, the second training module 1008 performs a second training operation on the second control model based on the first image taken by the first camera and the third image taken by the second camera to obtain a target control model. By using the first image and the third image with a changed viewpoint, the second control model is trained again, enabling the target control model to adapt to environments with different viewpoints. The present invention first uses a first image and a second image to perform a first training operation on a first control model to obtain a second control model. Then, it uses a third image with a changed viewpoint and the first image to perform a second training operation on the second control model to obtain a target control model. This allows the target control model to be applied to a specific environment and to adapt to an environment with a changed viewpoint, thereby giving the target control model a certain generalization ability.

[0119] In some embodiments, optionally, the first training module 1004 is used to initialize a first control model, wherein the first control model includes at least one of a fusion feature network, an action generation network, an evaluation network, a target evaluation network, or an inverse dynamics network; determine a fusion feature set based on a first image, a second image, and the first control model; determine a first control signal based on the fusion feature set and the first control model; control the robot to execute the first control signal, and calculate a feedback signal based on the executed robot; update the first control model based on the fusion feature set, the first control signal, and the feedback signal to obtain a third control model; re-acquire the first image and the second image, and input them into the third control model for retraining until the training count reaches a preset number of training times, after which the third control model is used as the second control model.

[0120] In this embodiment, the first training module 1004 is used to initialize the parameters in the first control model. The first control model includes at least one of a fusion feature network, an action generation network, an evaluation network, a target evaluation network, or an inverse dynamics network. The fusion feature network obtains the robot's fusion features based on the first and second images. The action generation (Actor) network outputs action values ​​based on the current state, i.e., it is responsible for generating the robot's control signals. The evaluation (Critic) network evaluates the actions selected by the Actor network, i.e., it predicts future cumulative rewards based on the current state and actions. The target evaluation (Critic) network provides stable target values, and the inverse dynamics network enables precise control. After initializing the first control model, the first and second images are input into it. The image features of the first and second images are fused in the first control model to obtain a fusion feature set, which allows for the acquisition of more comprehensive environmental information. Then, a first control signal is obtained based on the fusion feature set and the first control model. Finally, the robot's control actions are obtained in the first control model based on the fusion feature set, i.e., the robot's environmental information. Then, the robot executes corresponding control actions based on the first control signal, and calculates feedback signals based on the robot's current state and the object's position information after the control actions are completed. The first control model is then updated based on the fused feature set, the first control signal, and the feedback signal to obtain the third control model. This completes one training iteration. Finally, the first and second images are re-acquired, and then input into the third control model, repeating the above steps until the preset number of training iterations is reached (e.g., 1 million). The final third control model is then used as the second control model. By training the first control model to obtain the second control model, the second control model can exhibit good performance in the featured environment.

[0121] In some embodiments, optionally, the first training module 1004 is specifically configured to determine a first feature vector based on a first image; determine a second feature vector based on a second image; determine a first attention feature and a second attention feature based on the first feature vector, the second feature vector, and a fusion feature network, wherein the fusion feature network includes two convolutional neural networks and two sub-attention networks; and determine a fusion feature set based on the first attention feature and the second attention feature.

[0122] In this embodiment, the first training module 1004 is specifically used to determine a first feature vector of the first image based on the first image, and to determine a second feature vector of the second image based on the second image. The fusion feature network includes two CNN (Convolutional Neural Networks) encoding networks, two subsequent fully connected networks, and two sub-attention networks. First, the first image and the second image are respectively input into the two CNN encoding networks to obtain the first feature vector of the first image and the second feature vector of the second image. Then, the first feature vector is used as a query, and the second feature vector is used as a key-value pair (key and value) in one sub-attention network. This is then passed through two subsequent fully connected networks to obtain the first attention feature. The second feature vector is then used as a query, and the first feature vector is used as a key-value pair (key and value) in another sub-attention network. This is then passed through two subsequent fully connected networks to obtain the second attention feature. Finally, the first attention feature and the second attention feature are added together to obtain the fusion feature set. The present invention employs a visual information fusion method with a cross-view attention mechanism, which enables the extraction of image features obtained from a third-person perspective to process and pay attention to information from a first-person perspective, thereby allowing information from two different perspectives to interact and integrate, and jointly influence the downstream robot's operation strategy.

[0123] In some embodiments, optionally, the first training module 1004 is further configured to determine a first fusion feature of the current frame based on the fusion feature set; determine the mean vector and variance vector of the robot's control strategy based on the first fusion feature and the action generation network; establish a multidimensional Gaussian distribution of the control strategy based on the mean vector and variance vector; and determine a first control signal based on the multidimensional Gaussian distribution.

[0124] In this embodiment, the first training module 1004 is further specifically used to obtain the first fusion feature of the current frame based on the fusion feature set, and then input the first fusion feature into the action generation network to obtain the mean vector and variance vector of the robot control strategy. Then, the mean vector and variance vector are used to construct a multidimensional Gaussian distribution of the control strategy, and finally, the first control signal is obtained by sampling from the multidimensional Gaussian distribution. By using the action generation network to determine the first control signal, the accuracy of the first control signal is improved.

[0125] In some embodiments, optionally, the first training module 1004 is further configured to update the fusion feature network and the inverse dynamics network according to the fusion feature set and the first control signal; update the evaluation network and the fusion feature network according to the fusion feature set and the feedback signal; update the action generation network according to the updated evaluation network and the fusion feature set; and obtain a third control model based on the target evaluation network, the updated fusion feature network, the updated inverse dynamics network, the updated evaluation network, and the updated action generation network.

[0126] In this embodiment, the first training module 1004 is further used to update the parameters in the fusion feature network and the inverse dynamics network using the fusion feature set and the first control signal, so that the outputs of the fusion feature network and the inverse dynamics network are closer to the expectation. Then, the evaluation network and the target evaluation network are updated using the fusion feature set and the calculated feedback signal, and the fusion feature network is updated a second time. At the same time, the action generation network is updated using the updated evaluation network and the fusion feature set. Finally, the updated multiple networks are combined, that is, a third control model is generated based on the target evaluation network, the updated fusion feature network, the updated inverse dynamics network, the updated evaluation network, and the updated action generation network. The third control model is obtained by updating the first control model, so that the third control model can approach the control model that meets the requirements.

[0127] In some embodiments, the first training module 1004 is optionally further configured to determine a second fusion feature for the next frame based on the fusion feature set; determine a predicted second control signal based on the first fusion feature, the second fusion feature, and the inverse dynamics network; calculate a first error between the second control signal and the first control signal; and update the fusion feature network and the inverse dynamics network based on the first error.

[0128] In this embodiment, the first training module 1004 is further specifically used to determine the second fusion feature of the next frame from the fusion feature set, then connect the first and second fusion features and put them into the inverse dynamics network to predict the current robot action, i.e., the second control signal. The first control signal, i.e., the robot's actual executed action, is then used as the target value. The first error between the second control signal and the target value is then calculated, i.e., the prediction error of the inverse dynamics network. Finally, based on the prediction error of the inverse dynamics network, the parameters in the inverse dynamics network and the parameters of the fusion feature network are updated using gradient descent to obtain the updated inverse dynamics network and the updated fusion feature network. By calculating the difference between the second control signal predicted by the inverse dynamics network and the first control signal, the update of the inverse dynamics network is achieved.

[0129] In some embodiments, the first training module 1004 is optionally further configured to obtain a target value estimation function based on the second fusion feature, the feedback signal, and the target evaluation network; obtain a first loss function based on the first fusion feature, the evaluation network, and the target value estimation function; and update the fusion feature network and the evaluation network based on the first loss function.

[0130] In this embodiment, the first training module 1004 is further specifically used to input the second fused feature into the target evaluation network, and then combine it with the feedback signal obtained after the robot performs the corresponding action according to the first control signal to obtain the target value estimation function. Then, the first fused feature and the target value estimation function are input into the evaluation network to calculate the value function estimate and loss, i.e., the first loss function. Finally, the fused feature network, the evaluation network, and the target evaluation network are updated according to the first loss function.

[0131] In some embodiments, optionally, the first training module 1004 is further configured to determine the robot's control policy distribution based on the first fusion features and the action generation network; determine an estimate of the current control policy based on the updated evaluation network and the control policy distribution; determine a second loss function based on the estimate; and update the action generation network based on the second loss function.

[0132] In this embodiment, the first training module 1004 is further specifically used to input the first fused features into the action generation network, calculate the robot control policy distribution, then input the calculated control policy distribution into the updated evaluation network to obtain the estimated value of the current control policy, then calculate the loss of the action generation network, i.e., the second loss function, based on the estimated value, and finally update the action generation network using gradient descent based on the second loss function to obtain the updated action generation network.

[0133] In some embodiments, optionally, the second training module 1008 is used to freeze the action generation network and inverse dynamics network in the second control model; perform a first training operation on the frozen second control model according to the first image and the third image, and update the fusion feature network in the second control model to obtain a fourth control model; re-acquire the first image and the third image, and input them into the fourth control model for retraining until the training number reaches a preset number of training times, and then use the fourth control model as the target control model.

[0134] In this embodiment, the second training module 1008 is used to freeze the action generation network and inverse dynamics network in the second control model, that is, to keep the parameters in the action generation network and inverse dynamics network of the second control model unchanged. Then, the first training operation is performed again on the second control model using the first image and the third image, that is, the first training operation steps are re-executed using the first image and the third image. Since the action generation network and the inverse dynamics network are frozen, only the fusion feature network in the second control model is updated during the training process. Then, the fourth control model is obtained based on the updated fusion feature network, the frozen action generation network, and the frozen inverse dynamics network. Finally, the first image and the third image are re-acquired and input into the fourth control model for retraining until the training times reach a preset number of times, where the preset number of training times can be 1 million times. The fourth control model is then used as the target control model. By training the second control model using the third image and the first image after the viewpoint change, and by updating only the fusion feature network during the training process, it is ensured that the target control model can perform well in environments with different viewpoints.

[0135] Figure 11 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown; wherein, the electronic device 110 includes a memory 1102, a processor 1104, and a computer program stored in the memory 1102 and executable on the processor 1104, wherein the processor 1104 executes the computer program to implement the steps of the training method of the control model as described above.

[0136] The electronic device 110 provided by the present invention, when the processor 1104 executes the computer program, implements the steps of the above-described control model training method, which can achieve the technical effects of any of the above embodiments, and will not be described again.

[0137] One embodiment of the present invention provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the steps of the training method for the control model as described above.

[0138] The storage medium provided by this invention, when the computer program is executed by the processor, implements the steps of the training method of the above-described control model, and can achieve the technical effects of any of the above embodiments, which will not be repeated here.

[0139] One embodiment of the present invention provides a computer program product including a computer program that, when executed by a processor, implements the steps of a training method for a control model as described in any of the above embodiments.

[0140] The computer program product provided in this embodiment implements the steps of the training method for the control model as described in any embodiment of the present invention, and therefore has all the beneficial effects of the training method for the control model as described in any embodiment of the present invention, which will not be repeated here.

[0141] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance, unless otherwise expressly specified and limited. The terms "connection," "installation," and "fixing," etc., should be interpreted broadly. For example, "connection" can mean a fixed connection, a detachable connection, or an integral connection; it can mean a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0142] In the description of this specification, the terms "one embodiment," "some embodiments," "specific embodiment," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0143] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of training a control model, characterized by, The method comprises the following steps: collecting a first image captured by a first camera and a second image captured by a second camera at a first field of view, wherein the first camera is arranged on a robot, and the second camera is arranged at a preset distance from the robot and is fixedly arranged; performing a first training operation on a first control model according to the first image and the second image to obtain a second control model; collecting a third image captured by the second camera at a second field of view; performing a second training operation on the second control model according to the first image and the third image to obtain a target control model. 2.The method of claim 1, wherein, The step of performing a first training operation on a first control model according to the first image and the second image to obtain a second control model comprises the following steps: initializing the first control model, wherein the first control model comprises at least one of a fusion feature network, an action generation network, an evaluation network, a target evaluation network or an inverse dynamics network; determining a fusion feature set according to the first image, the second image and the first control model; determining a first control signal according to the fusion feature set and the first control model; controlling the robot to execute the first control signal, and calculating a feedback signal according to the executed robot; updating the first control model according to the fusion feature set, the first control signal and the feedback signal to obtain a third control model; recollecting the first image and the second image and inputting them into the third control model for retraining, and after the training times reach a preset training times, taking the third control model as the second control model.

3. The training method of a control model according to claim 2, wherein, The step of determining a fusion feature set according to the first image, the second image and the first control model comprises the following steps: determining a first feature vector according to the first image; determining a second feature vector according to the second image; determining a first attention feature and a second attention feature according to the first feature vector, the second feature vector and the fusion feature network, wherein the fusion feature network comprises two convolutional neural networks and two sub-attention networks; determining the fusion feature set according to the first attention feature and the second attention feature. 4.The method of Claim 2, wherein, The step of determining a first control signal according to the fusion feature set and the first control model comprises the following steps: determining a first fusion feature of a current frame according to the fusion feature set; determining a mean vector and a variance vector of a control strategy of the robot according to the first fusion feature and the action generation network; establishing a multi-dimensional Gaussian distribution of the control strategy according to the mean vector and the variance vector; determining the first control signal according to the multi-dimensional Gaussian distribution.

5. The training method of a control model according to claim 4, wherein, The step of updating the first control model according to the fusion feature set, the first control signal and the feedback signal to obtain a third control model comprises the following steps: updating the fusion feature network and the inverse dynamics network according to the fusion feature set and the first control signal; updating the evaluation network and the fusion feature network according to the fusion feature set and the feedback signal; updating the action generation network according to the updated evaluation network and the fusion feature set; obtaining the third control model according to the target evaluation network, the updated fusion feature network, the updated inverse dynamics network, the updated evaluation network and the updated action generation network. 6.The method of Claim 5, wherein, The step of updating the fusion feature network and the inverse dynamics network according to the fusion feature set and the first control signal comprises: determining a second fusion feature of a next frame according to the fusion feature set; determining a predicted second control signal according to the first fusion feature, the second fusion feature and the inverse dynamics network; calculating a first error between the second control signal and the first control signal; updating the fusion feature network and the inverse dynamics network respectively according to the first error.

7. The training method of a control model according to claim 6, wherein, The step of updating the evaluation network and the fusion feature network according to the fusion feature set and the feedback signal comprises: obtaining a target value estimation function according to the second fusion feature, the feedback signal and the target evaluation network; obtaining a first loss function according to the first fusion feature, the evaluation network and the target value estimation function; updating the fusion feature network and the evaluation network respectively according to the first loss function. 8.The method of Claim 7, wherein, The step of updating the action generation network according to the updated evaluation network and the fusion feature set comprises: determining a control policy distribution of the robot according to the first fusion feature and the action generation network; determining an estimated value of the current control policy according to the updated evaluation network and the control policy distribution; determining a second loss function according to the estimated value; updating the action generation network according to the second loss function. 9.The method of Claim 1 to 8, wherein, The step of performing a second training operation on the second control model according to the first image and the third image to obtain a target control model comprises: freezing the action generation network and the inverse dynamics network in the second control model; performing a first training operation on the frozen second control model according to the first image and the third image, and updating the fusion feature network in the second control model to obtain a fourth control model; recollecting the first image and the third image and inputting them into the fourth control model for retraining, until the training times reach a preset training times, and taking the fourth control model as the target control model.

10. A training system of a control model, characterized by, comprises: a first acquisition module, the first acquisition module is used for acquiring a first image shot by a first camera and a second image shot by a second camera in a first field of view, wherein the first camera is arranged on a robot, the second camera is a preset distance away from the robot and is fixedly arranged; a first training module, the first training module is used for performing a first training operation on a first control model according to the first image and the second image to obtain a second control model; a second acquisition module, the second acquisition module is used for acquiring a third image shot by the second camera in a second field of view; A second training module is configured to perform a second training operation on the second control model according to the first image and the third image, to obtain a target control model.

11. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the training method of the control model according to any one of claims 1 to 9 when executing the computer program.

12. A storage medium having stored thereon a computer program, characterized in that The computer program, when executed by the processor, implements the steps of the training method of the control model according to any one of claims 1 to 9.

13. A computer program product comprising computer instructions, characterized in that, The computer program, when executed by the processor, implements the steps of the training method of the control model according to any one of claims 1 to 9. The computer program, when executed by the processor, implements the steps of the training method of the control model according to any one of claims 1 to 9.

Images