Mechanical arm control method and device based on pre-training model, equipment and medium

Abstract

The application provides a mechanical arm control method, device and equipment based on a pre-training model and a medium, and relates to the technical field of mechanical arm control.The mechanical arm control method comprises the following steps: S1, a plurality of pictures of different perspectives of a mechanical arm in a current environment state are acquired, and the plurality of pictures of different perspectives are input into a set feature extractor for processing to obtain a plurality of feature maps of different perspectives; S2, after the plurality of feature maps of different perspectives are flattened, the flattened feature maps are spliced through a randomly initialized fully connected network layer to obtain a visual feature vector; S3, according to the visual feature vector, a deep deterministic policy gradient model is used to predict a next action of the mechanical arm; and S4, the mechanical arm is controlled to execute according to the predicted next action of the mechanical arm.The application combines deep deterministic policy gradient model reinforcement learning, and the model has a high convergence speed and a high success rate, and the efficiency of learning control of the mechanical arm is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical arm control, in particular to a mechanical arm control method and device based on a pre-trained model, equipment and medium. BACKGROUND

[0002] In the field of modern robotics, visual input mechanical arm control reinforcement learning algorithms based on pre-trained models have been widely applied. Reinforcement learning is a method of learning optimal strategies through interaction with the environment, and the combination of visual input based on pre-trained models can enable the mechanical arm to better perceive the surrounding environment, thereby learning more intelligent and adaptive control strategies. This control algorithm not only improves the control accuracy and efficiency of the mechanical arm, but also adapts to different working scenarios and task requirements, and has important practical application value.

[0003] However, the current visual input mechanical arm control reinforcement learning algorithm based on pre-trained models still faces some challenges. For example, how to effectively combine the knowledge of pre-trained models with the reinforcement learning framework, how to handle the complexity of visual input data and environmental noise interference, and how to solve the instability and convergence difficulty of model training, etc. all need further research and solution.

[0004] Therefore, after studying the existing technology, the applicant proposes the present application. SUMMARY

[0005] The present application aims to provide a mechanical arm control method, device, equipment and medium based on a pre-trained model to solve the problems of instability and convergence difficulty of visual input mechanical arm control reinforcement learning model training in existing methods.

[0006] To solve the above technical problems, the present application realizes the following technical scheme:

[0007] A mechanical arm control method based on a pre-trained model, comprising:

[0008] S1, obtaining multiple pictures of different angles of the mechanical arm under the current environment state, and inputting the multiple pictures of different angles into a set feature extractor for processing to obtain multiple feature maps of different angles; wherein the multiple pictures of different angles complement each other to ensure that the mechanical arm is not blocked at the same time, and a complete and accurate mechanical arm model is constructed;

[0009] S2, flattening the multiple feature maps of different angles, and then splicing them through a randomly initialized fully connected network layer to obtain a visual feature vector;

[0010] S3, according to the visual feature vector, a deep deterministic policy gradient model is used to predict the next action of the robot arm; wherein the deep deterministic policy gradient model includes an Actor network for learning a policy function and outputting an action;

[0011] S4, according to the predicted next action of the robot arm, the robot arm is controlled to execute.

[0012] The application also provides a robot arm control device based on a pre-trained model, comprising:

[0013] A feature extraction unit is configured to obtain a feature map of an input image, acquire multiple images of the robot arm at different angles in a current environment state, and input the multiple images at different angles into a set feature extractor to obtain multiple feature maps at different angles; wherein the multiple images at different angles complement each other to ensure that the robot arm is not blocked at the same time, and a complete and accurate robot arm model is constructed.

[0014] A full connection unit is configured to obtain a visual feature vector, and the multiple feature maps at different angles are flattened through a flatten and spliced through a randomly initialized full connection network layer to obtain the visual feature vector.

[0015] A policy prediction unit is configured to use a deep deterministic policy gradient model to predict the next action of the robot arm according to the visual feature vector; wherein the deep deterministic policy gradient model includes an Actor network for learning a policy function and outputting an action.

[0016] An execution unit is configured to control the robot arm to execute according to the predicted next action of the robot arm.

[0017] The application also provides a robot arm control device based on a pre-trained model, which is applied to a robot arm and comprises a processor and a memory, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the robot arm control method based on a pre-trained model.

[0018] The application also provides a computer readable storage medium, wherein the computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by a processor of a device in which the computer readable storage medium is located to implement the robot arm control method based on a pre-trained model.

[0019] Compared with the prior art, the application has the following beneficial effects:

[0020] The pre-training model-based visual input mechanical arm control reinforcement learning algorithm can better utilize the prior knowledge of the pre-training model when processing visual input data, and improve the learning efficiency and control accuracy of the mechanical arm.

[0021] In addition, the pre-training model with different depth hidden layers, and the excessive or insufficient number of convolution layers and residual blocks in the model will affect the final success rate and convergence speed of the model, so the appropriate convolution network should be selected as the pre-training model according to the actual situation. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, and it should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0023] Figure 1 A schematic diagram of a pre-training model-based mechanical arm control method provided for the first embodiment of the present application.

[0024] Figure 2 A model structure schematic diagram of a pre-training model-based mechanical arm control method provided for the first embodiment of the present application.

[0025] Figure 3 A Reach task side view diagram provided for the first embodiment of the present application.

[0026] Figure 4 Another side view diagram of the Reach task provided for the first embodiment of the present application.

[0027] Figure 5 A pre-training feature extractor and reinforcement learning migration process diagram provided for the first embodiment of the present application.

[0028] Figure 6 A success rate change diagram of the pre-training visual features obtained by different pre-training algorithms for the Reach task provided for the first embodiment of the present application.

[0029] Figure 7 A success rate change diagram of the pre-training visual features obtained by different pre-training algorithms for the Push task provided for the first embodiment of the present application.

[0030] Figure 8 Success rate of the output of different depth hidden layers as pre-trained visual features for the Reach task as a function of the number of steps.

[0031] Figure 9 A schematic diagram of a robot arm control device based on a pre-trained model is provided in Embodiment Two of the present application.

[0032] The present application will be further described below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0034] Embodiment One

[0035] Embodiment One of the present application provides a robot arm control method based on a pre-trained model, which can be implemented by a robot arm control device based on a pre-trained model (hereinafter referred to as a control device), and in particular, by one or more processors in the control device.

[0036] In this embodiment, the control device can be an electronic device equipped with a processor, the processor has a computer program of the robot arm control method based on a pre-trained model and the computer program can be executed, such as a computer, a smart phone, a smart tablet, etc.

[0037] Traditional reinforcement learning methods often face the problems of unstable training and difficult convergence when processing visual input data, so the present application proposes an algorithm combining visual feature extraction of a pre-trained model and reinforcement learning. The algorithm uses the rich features learned by the pre-trained model in visual tasks to assist the robot arm in accurate perception and precise control in complex environments. Normally, the depth of the convolutional network used is several or even dozens of times that of previous work. Our model is only pre-trained on external visual datasets, and in the case of using unprocessed raw visual input, our pre-trained features can compete with real features in task success rate, and the convergence speed is faster.

[0038] In traditional robot control algorithms, many studies have shown that different hidden layers in deep convolutional networks extract different kinds of features. The initial hidden layers tend to extract low-level features such as edges, shapes, and patterns from the original input data. Deeper hidden layers learn to combine these low-level features into more abstract and high-level representations that are very useful for specific tasks.

[0039] As shown in Figure 1 and Figure 2 , a robot arm control method based on a pre-trained model includes steps S1-S4.

[0040] S1, obtain multiple pictures of the robot arm from different angles under the current environment state, and input the multiple pictures from different angles into a set feature extractor for processing to obtain feature maps of multiple angles; wherein the multiple pictures from different angles complement each other to ensure that the robot arm is not blocked at the same time, and a complete and accurate robot arm model is constructed.

[0041] Further, multiple pictures of the robot arm from different angles are obtained to ensure that the robot arm in the multiple pictures is not blocked at the same time, and the multiple pictures complement each other, so that a complete robot arm under the current environment state can be obtained after inputting the model. As shown in Figure 3 and Figure 4 In this embodiment, we choose to input the feature extractor with a side view of the robot arm and another side view of the robot arm. Other angle pictures can also be selected, which will not be described here. These pictures are represented by a [3, 256, 256] tensor. After processing by the feature extractor, we get the feature maps of the two angles.

[0042] As shown in Figure 5 , further, the feature extractor is a convolutional network layer of a pre-trained visual model, and the training steps include A1-A3.

[0043] A1, obtain an image data set and pre-process the image data set;

[0044] A2, input the pre-processed image data set into a set pre-trained visual model, extract features of the pre-processed image data through the convolutional network layer of the pre-trained visual model, obtain feature vectors of the image to capture local features and semantic information in the image, and use a set pre-training algorithm to train the pre-trained visual model to obtain a trained visual model; wherein the pre-training algorithm includes a supervised learning method and an unsupervised learning method;

[0045] A3, freeze the parameters of the convolutional network layer of the trained visual model to obtain a feature extractor.

[0046] The purpose of steps A1-A3 is to apply the visual feature extraction capabilities learned by the pre-trained model on large-scale datasets to our robot control task. Convolutional networks have strong feature extraction capabilities in image processing, effectively capturing local features and semantic information in images, so we choose to cut off the convolutional network part as the feature extractor.

[0047] The reason for cutting off the convolutional network part of the pre-trained model without including the fully connected network layer is that the convolutional network is mainly responsible for extracting feature representations of images, while the fully connected network layer is used to map these features to specific class or task spaces. In our robot control task, we do not need to perform specific class classification on images, but rather use the extracted feature representations as input to help the robot understand the environment and make appropriate control decisions. Therefore, we choose to only cut off the convolutional network part as the feature extractor to reduce the complexity of the model and improve computational efficiency, while retaining the advantages of the pre-trained model in image feature extraction.

[0048] S2, the feature maps of multiple perspectives are flattened and then concatenated through a randomly initialized fully connected network layer to obtain a visual feature vector.

[0049] In this embodiment, the feature maps of multiple perspectives are concatenated into a visual feature vector that contains vectors of information such as robot position and velocity, object position, and target position, which can completely describe the state of the environment. We directly use this vector to train the policy network.

[0050] In particular, since using pictures as input to the visual model would cause huge storage pressure, in the actual code, we use the pre-trained model as part of the environment rather than the model to reduce storage pressure.

[0051] The parameters of the feature extractor are frozen and concatenated with the fully connected network as input to the deep deterministic policy gradient model. The purpose of this step is to extract image features using the pre-trained model while keeping the weights of the feature extractor unchanged, i.e., freezing the parameters, to ensure that these parameters are not modified during subsequent training. The main considerations are as follows:

[0052] First, the pre-trained model has been fully trained on large-scale datasets and has learned effective methods for extracting image features, so the parameters of the feature extractor can be considered as high-quality feature representations. We do not want to randomly modify these parameters during subsequent training to avoid losing the rich information contained therein.

[0053] Secondly, freezing the parameters of the feature extractor can effectively reduce the number of training parameters of the model, reducing the complexity and computational cost of training. In reinforcement learning tasks, we usually face the problem of small data volume and long training time, so by freezing the parameters of the feature extractor, we can effectively improve the training efficiency, speed up the convergence speed of the model, and more easily avoid the problem of overfitting.

[0054] S3, according to the visual feature vector, a deep deterministic policy gradient model is used to predict the next action of the robot arm; wherein the deep deterministic policy gradient model includes an Actor network for learning a policy function and outputting an action.

[0055] Deep Deterministic Policy Gradient (DDPG) is a reinforcement learning algorithm for continuous action space, which combines deep neural networks and deterministic policy gradient methods, and can effectively handle high-dimensional and continuous action space problems.

[0056] The core idea of the DDPG model algorithm is to use two neural networks: one is the Actor network, which is used to learn the policy function and directly output the action; the other is the Critic network, which is used to learn the value function and evaluate the value of the action taken in the current state. Through the Actor-Critic structure, the DDPG algorithm can learn the policy and value function at the same time, and can handle continuous action space problems.

[0057] In this embodiment, the method of the present application receives environmental state information and selects an action according to the current policy. Then, the action is executed, and the feedback of the environment is observed, including the next state and the corresponding reward signal.

[0058] The environmental state information is represented in the form of a vector, including the position and velocity of the robot arm, the position of the object, the target position, etc. These information is encoded into a vector of length 23 to describe the complete state of the environment. In the model trained with visual input, the state information can be represented by color pictures from the side and top view, and after the feature extractor, the feature map is obtained, and finally converted into a feature vector.

[0059] The visual feature vector is output by the Actor network to predict the next action of the robot arm.

[0060] S4, according to the predicted next action of the robot arm, the robot arm is controlled to execute.

[0061] Further, after the mechanical arm performs an action, the Critic network is used to judge and evaluate whether the current action successfully reaches the target position, output a reward signal, record and save the current environment state, the current action, the reward signal and the next step action of the mechanical arm, wherein the current environment state includes the position of the mechanical arm, the speed of the mechanical arm, the position of the object and the target position.

[0062] The deep deterministic policy gradient model is trained according to the stored current environment state, current action, reward signal and next step action of the mechanical arm, and the parameters are continuously updated until a preset stopping condition is reached.

[0063] The training process of the DDPG algorithm includes the following steps:

[0064] Step 1: initialize the parameters of the Actor and Critic networks;

[0065] Step 2: continuously obtain the current environment state, control the mechanical arm to move, and collect experience data;

[0066] Step 3: store the experience data in the experience replay buffer;

[0067] Step 4: randomly sample a batch of data from the experience replay buffer;

[0068] Step 5: update the parameters of the Actor and Critic networks using the sampled data;

[0069] Repeat steps 2 to 5 until the stopping condition is reached.

[0070] Through such a training process, the DDPG algorithm can learn an excellent policy function and value function, thereby achieving efficient exploration and optimization of continuous action space.

[0071] Further, the parameter update of the Actor network follows the policy gradient method to maximize the action value function, and the parameter update formula of the Actor network is as follows:

[0072]

[0073] wherein, represents the derivative of the expected value of the reward with respect to the parameter θ μ ; J represents the target function of the policy; D represents the experience replay buffer; represents the expected value of sampling the state s in the experience replay buffer ; Q(s, μ(s| θ μ )) represents the value estimate provided by the Critic network for selecting the action μ(s| θ μ ) under the state s; μ(s| θμ ) represents the probability of selecting action a in state s output by the Actor network μ

[0074] The parameter update of the Critic network follows the minimization of the TD error, which makes it approximate the true value of the action-value function. The objective function of the Critic network is:

[0075]

[0076] where θ Q represents the parameters of the Critic network; D represents the experience replay buffer, represents the expected value of the four-tuple (s, a, r, s') sampled from the experience replay buffer D; Q(s, a | θ Q ) represents the action-value function of the Critic network for action a in state s; a represents an action; y = r + γQ(s', μ(s'| θ μ | θ Q ), represents the target value of TD; γ is the discount factor, and r represents the current reward value.

[0077] This process usually uses gradient descent algorithms, such as the Adam optimizer, to minimize the loss function to improve the performance of the algorithm.

[0078] The algorithm of the present application continuously interacts with the environment and updates the parameters until the preset stopping condition is reached. The entire process is an iterative loop, and the algorithm continuously interacts with the environment and learns, gradually improving its performance and adapting to changes in the environment.

[0079] In another specific embodiment, in order to establish a simulation environment for the robotic arm, we choose the Panda Gym library as our simulation environment, which is developed based on the PyBullet simulator. In this environment, our invented algorithm needs to control a Franka Emika Panda robotic arm with a gripper to complete various tasks. This robotic arm has 7 degrees of freedom, and its end effector is a parallel gripper that is blocked when the task does not require it. We tested it on the Reach and Push tasks in the Panda Gym library.

[0080] In the Reach task, the present invention needs to control the end effector of the robotic arm to reach a randomly generated target position, which is located in a 30cm x 30cm x 30cm space. During the training process, the environment of the Reach task is deployed on four cores of the CPU, and the algorithm of the present invention interacts with the environment a total of 10000 times.​

[0081] As shown in Figure 6 and Figure 7 , we use the pre-training visual features obtained by supervised learning and unsupervised learning pre-training algorithm for Reach / Push task. Compared with the random initialization algorithm, the pre-training algorithm can significantly reduce the number of steps required to approach the true value, has fast convergence speed, high learning efficiency, and the supervised learning training effect is faster than the unsupervised learning convergence speed, and has obvious advantages in learning efficiency.

[0082] As shown in Figure 8 and Table 1, the comparison of the success rate curve of the model output with different depth hidden layers as pre-training visual features for Reach task shows that the model with appropriate number of residual blocks and convolution layers should be used for training, and too many or too few hidden layers will affect the final success rate and convergence speed of the model. For this embodiment, when the number of convolution layers is 41 and the number of residual blocks is 13, the model learning effect is best.

[0083] Table 1 Different depth hidden layers for residual block and convolution layer number and final success rate of model

[0084] Type Number of convolution layers Number of residual blocks Final success rate Backbone 1 11 3 0.08 Backbone 2 23 7 0.99 Backbone 3 41 13 1.00 Backbone 4 50 16 0.07

[0085] Compared with the prior art, the present application has the following advantages:

[0086] The visual input robot arm control reinforcement learning algorithm based on the pre-training model can better utilize the prior knowledge of the pre-training model when processing visual input data, and improve the learning efficiency and control accuracy of the robot arm. The visual feature extraction method of the pre-training model combined with the deep deterministic policy gradient model reinforcement learning has fast model convergence speed and high success rate, effectively improves the learning control efficiency of the robot arm, and provides an effective solution for the autonomous control of the robot arm in complex environment.

[0087] In addition, the pre-training model with different depth hidden layers, too many or too few convolution layers and residual blocks in the model will affect the final success rate and convergence speed of the model, so the appropriate convolution network should be selected as the pre-training model according to the actual situation.

[0088] Embodiment two

[0089] As shown in Figure 9 , the second embodiment of the present application further provides a robot arm control device based on a pre-training model, which comprises:

[0090] The feature extraction unit is configured to obtain a feature map of the input image, acquire multiple images of different perspectives of the robot arm in the current environment state, input the multiple images of different perspectives into a set feature extractor for processing, and obtain multiple perspective feature maps; wherein the multiple images of different perspectives complement each other to ensure that the robot arm is not blocked at the same time, and a complete and accurate robot arm model is constructed.

[0091] The full connection unit is configured to obtain a spliced visual feature vector, flatten the multiple perspective feature maps, splice the flattened multiple perspective feature maps through a randomly initialized full connection network layer, and obtain the visual feature vector.

[0092] The policy prediction unit is configured to predict a next action of the robot arm according to the visual feature vector by using a deep deterministic policy gradient model, wherein the deep deterministic policy gradient model comprises an Actor network configured to learn a policy function and output an action.

[0093] The execution unit is configured to control the robot arm to perform according to the predicted next action of the robot arm.

[0094] Embodiment three

[0095] The third embodiment of the present application further provides a robot arm control device based on a pre-trained model, applied to a robot arm, comprising a memory and a processor, the memory stores a computer program, and the computer program can be executed by the processor to implement the robot arm control method based on the pre-trained model as described above.

[0096] Embodiment four

[0097] The fourth embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor of the device where the computer readable storage medium is located to implement the robot arm control method based on the pre-trained model as described above.

[0098] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed apparatus and method can also be implemented by other manners. The apparatus and method embodiments described above are only illustrative, for example, the flowchart in the drawings shows the possible implementation architecture, function and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code, which includes one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementation manners, the functions noted in the blocks can also occur in different order from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for executing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0099] In addition, each functional module in the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0100] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes. It should be noted that in this document, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed, or include elements inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device that includes the element.

[0101] The terminology used in the description of the embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the description of the embodiments and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0102] It should be understood that the term "and / or" as used herein merely describes associated objects, and can exist in three forms, for example, A and / or B can mean that A exists alone, A and B exist together, or B exists alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.

[0103] Depending on context, the word "if" as used herein can be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]."

[0104] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order of the objects. Understandably, the "first\second" can be interchanged in a specific order or sequence as allowed. It should be understood that the objects distinguished by "first\second" can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0105] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A robotic arm control method based on a pre-trained model, characterized in that, include: S1. Acquire multiple images of the robotic arm from different perspectives under the current environmental conditions, and input the multiple images from different perspectives into the set feature extractor for processing to obtain feature maps from multiple perspectives; wherein, the multiple images from different perspectives complement each other to ensure that the robotic arm is not occluded at the same time, and construct a complete and accurate robotic arm model. S2, after flattening the feature maps from multiple perspectives, they are concatenated through a randomly initialized fully connected network layer to obtain a visual feature vector; S3, Based on the visual feature vector, a deep deterministic policy gradient model is used to predict the next action of the robotic arm; wherein the deep deterministic policy gradient model includes an Actor network for learning the policy function and outputting the action. S4, based on the predicted next action of the robotic arm, control the robotic arm to execute; The feature extractor is a convolutional network layer of a pre-trained visual model, and its training steps are as follows: A1. Obtain the image dataset and preprocess it. A2. Input the preprocessed image dataset into the pre-trained visual model, extract features from the preprocessed image data through the convolutional network layer of the pre-trained visual model to obtain the feature vector of the image, so as to capture the local features and semantic information in the image, and train the pre-trained visual model using the pre-training algorithm to obtain the trained visual model; wherein, the pre-training algorithm includes supervised learning method and unsupervised learning method. A3. Freeze the parameters of the convolutional network layer of the trained visual model to obtain the feature extractor.

2. The robotic arm control method based on a pre-trained model according to claim 1, characterized in that... The deep deterministic policy gradient model also includes a Critic network for learning value functions and evaluating the quality of policies.

3. The robotic arm control method based on a pre-trained model according to claim 2, characterized in that... After the robotic arm performs an action, the process also includes: judging and evaluating whether the current action has successfully reached the target position through the Critic network, outputting a reward signal, and recording and saving the current environmental state, the current action, the reward signal, and the next action of the robotic arm. The current environmental state includes the position of the robotic arm, the speed of the robotic arm, the position of the object, and the target position.

4. The robotic arm control method based on a pre-trained model according to claim 3, characterized in that... The deep deterministic policy gradient model is trained based on the stored current environment state, current action, reward signal, and the next action of the robotic arm, and its parameters are continuously updated until a preset stopping condition is reached; wherein, The parameter updates of the Actor network follow the policy gradient method to maximize the action value function. The parameter update formula for the Actor network is as follows: ； in, The expected value of the reward is expressed as a function of the parameter. The derivative of ; J represents the objective function of the policy; This represents the experience replay buffer; Indicates the experience replay buffer The expected value of sampling from state s in the data; This indicates the action selection provided by the Critic network in state s. Value estimation; This represents the action selected by the Actor network in a given state s. The probability of; The parameter updates of the Critic network follow the principle of minimizing the TD error to approximate the true value of the action value function. The objective function of the Critic network is: ； in, These represent the parameters of the Critic network; This represents the sampled state s, action a, current reward value r, and next state from the experience replay buffer. Quadruple Expected value; This represents the action value function of the Critic network for action a in state s; , where represents the target value of TD; γ is the discount factor.

5. A robotic arm control device based on a pre-trained model, characterized in that, include: The feature extraction unit is used to obtain feature maps of the input image, acquire multiple images of the robotic arm from different perspectives in the current environment, and input these images into a pre-defined feature extractor for processing to obtain feature maps from multiple perspectives. These images from different perspectives complement each other to ensure that the robotic arm is not simultaneously occluded, thus constructing a complete and accurate robotic arm model. The feature extractor is a convolutional network layer of a pre-trained visual model, and its training steps are as follows: A1. Obtain the image dataset and preprocess it. A2. Input the preprocessed image dataset into the pre-trained visual model, extract features from the preprocessed image data through the convolutional network layer of the pre-trained visual model to obtain the feature vector of the image, so as to capture the local features and semantic information in the image, and train the pre-trained visual model using the pre-training algorithm to obtain the trained visual model; wherein, the pre-training algorithm includes supervised learning method and unsupervised learning method. A3. Freeze the parameters of the convolutional network layer of the trained visual model to obtain the feature extractor; The fully connected unit is used to obtain the stitched visual feature vector. The feature maps from multiple perspectives are flattened and then stitched together through a randomly initialized fully connected network layer to obtain the visual feature vector. The policy prediction unit is used to predict the next action of the robotic arm based on the visual feature vector and using a deep deterministic policy gradient model; wherein the deep deterministic policy gradient model includes an Actor network for learning policy functions and outputting actions. The execution unit is used to control the robotic arm to perform actions based on the predicted next action of the robotic arm.

6. A robotic arm control device based on a pre-trained model, applied to a robotic arm, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that can be executed by the processor to implement a robotic arm control method based on a pre-trained model as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by the processor of the device on which the computer-readable storage medium resides, implement a robotic arm control method based on a pre-trained model as described in any one of claims 1-4.

Images