Mechanical arm force feedback control method based on deep reinforcement learning

Abstract

The invention discloses a mechanical arm force feedback control method based on deep reinforcement learning, and the method comprises the following steps: 1, data collection and preprocessing: collecting environment data, collecting operation data, and preprocessing the data; 2, designing a self-adaptive deep reinforcement learning algorithm, including initial model training, self-adaptive learning module and model online updating; 3, introducing a prediction model, including human operation intention prediction and motion trail prediction; and step 4, performance test and optimization, including simulation test and safety and efficiency evaluation. The force feedback control precision of the mechanical arm during operation in a complex environment and the safety and efficiency of man-machine cooperation are improved. According to the method, a control strategy is updated and adjusted in real time by designing an adaptive deep reinforcement learning algorithm and introducing a human operation intention prediction model, so that environmental changes and unknown interference can be better coped with.

Description

Technical Field

[0001] The present invention relates to the technical field of robotic arm control, and in particular to a robotic arm force feedback control method based on deep reinforcement learning. Background Art

[0002] The existing robotic arm control process usually combines multiple sensors (such as force sensors, position sensors, vision sensors, etc.) to perceive the environment and force feedback.

[0003] However, when the robotic arm operates in a complex environment, the force feedback control method needs to quickly adapt to changes and disturbances, and the traditional control method lacks flexibility. In addition, when the robotic arm collaborates with humans, how to ensure safety and efficiency is a challenge. Existing methods are difficult to achieve both efficiency and safety. To this end, we propose a robotic arm force feedback control method based on deep reinforcement learning to solve the above problems. Summary of the invention

[0004] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0005] In view of the above-mentioned problems existing in the existing robot arm force feedback control method based on deep reinforcement learning, the present invention is proposed.

[0006] Therefore, the purpose of the present invention is to provide a mechanical arm force feedback control method based on deep reinforcement learning, which improves the force feedback control accuracy and the safety and efficiency of human-machine collaboration when the mechanical arm is operating in a complex environment. This method realizes real-time updating and adjustment of control strategies by designing an adaptive deep reinforcement learning algorithm and introducing a human operation intention prediction model, so as to better cope with environmental changes and unknown interference.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a mechanical arm force feedback control method based on deep reinforcement learning, comprising the following steps:

[0008] Step 1: Data collection and preprocessing

[0009] Environmental data collection: Various data about the robot's surroundings, such as temperature, humidity, light, and object position, are collected in real time through a sensor network;

[0010] Operation data collection: collect motion data of human operators, including gestures, force feedback, operation trajectory and intention data;

[0011] Data preprocessing: De-noising, normalization and feature extraction of the collected data to form feature vectors for training deep reinforcement learning models;

[0012] Step 2: Adaptive Deep Reinforcement Learning Algorithm Design

[0013] Initial model training: Using the preprocessed data set, the deep reinforcement learning algorithm is used to train the initial control strategy of the robot arm;

[0014] Adaptive learning module: Design an adaptive learning module that can dynamically adjust the parameters of the deep reinforcement learning model and update the control strategy in real time according to changes in the real-time environment and operation data;

[0015] Model online update: Implement the online update function of the model, so that the robot arm can adapt to environmental changes and unknown interference through continuous learning and optimization during operation;

[0016] Step 3: Introduction of prediction model

[0017] Prediction of human operation intention: Use the convolutional neural network deep learning model to predict the gestures and movement trajectories of human operators and determine their operation intentions;

[0018] Motion trajectory prediction: Based on the collected human operation data, a prediction model is established to predict the motion trajectory of the robot arm, thereby optimizing its control strategy;

[0019] Step 4: Performance testing and optimization

[0020] Simulation testing: Test the performance of the control algorithm in a virtual simulation environment to verify its adaptability and stability in different scenarios and environments;

[0021] Safety and efficiency assessment: Evaluate the safety and operational efficiency of the robot arm in human-machine collaboration to ensure that it meets actual application requirements.

[0022] As a preferred solution of the mechanical arm force feedback control method based on deep reinforcement learning described in the present invention, the initial model training in step 2 uses a deep Q network as a deep reinforcement learning algorithm, and the specific formula is as follows:

[0023] Q function update:

[0024]

[0025] Among them, s represents the current state, a represents the current action, r represents the immediate reward, s′ represents the next state, α represents the learning rate, and γ represents the discount factor;

[0026] Use a neural network to approximate the Q function:

[0027] Q(s,a;θ)≈Q(s,a)

[0028] Among them, θ is the parameter of the neural network;

[0029] Loss function:

[0030]

[0031] Among them, θ - is the parameter of the objective function, and D is the sample set in the experience replay pool;

[0032] Parameter update:

[0033]

[0034] Target network parameter update:

[0035] θ - ←θ(periodic update).

[0036] As a preferred solution of the mechanical arm force feedback control method based on deep reinforcement learning described in the present invention, the adaptive learning module in step 2 is used to dynamically adjust the parameters of the deep reinforcement learning model to achieve real-time update control, including:

[0037] Real-time data collection: collect new samples from environment and operation data in real time and add them to the experience replay pool D;

[0038] Dynamically adjust the learning rate: Use the adaptive learning rate algorithm to adjust the learning rate of the neural network:

[0039]

[0040] Among them, α t is the learning rate at the tth iteration. The adaptive algorithm adaptively adjusts the learning rate based on the first-order moment estimate and the second-order moment estimate of the gradient:

[0041] m t =β1m t-1 +(1-β1)g t

[0042]

[0043] Among them, g t is the current gradient, β1 and β2 are exponential decay rates, ∈ is a decimal to prevent division by zero; m t is the first-order moment estimate of the gradient; v t is the second moment estimate of the gradient; is the bias correction of the first-order moment estimate; is the bias correction value of the second-order moment estimate; t is the current time step.

[0044] As a preferred solution of the mechanical arm force feedback control method based on deep reinforcement learning described in the present invention, wherein: the step 2 also includes experience replay and priority sampling process, introduces a priority experience replay mechanism, and prioritizes samples with larger sampling errors:

[0045]

[0046] Among them, P(i) is the probability of sample i being sampled, δ i is the TD error of sample i, w is the hyperparameter for adjusting the sampling bias;

[0047] The model is updated online by combining newly collected data with the samples from the experience replay pool with priority sampling, and training is performed regularly to update the model parameters θ:

[0048]

[0049] As a preferred solution of the mechanical arm force feedback control method based on deep reinforcement learning described in the present invention, wherein: the convolutional neural network used for gesture and motion trajectory prediction in step three includes:

[0050] (1) Network architecture design:

[0051] Input layer: input gesture and motion trajectory data, usually image sequence or time series;

[0052] Convolutional layer: extract spatial features;

[0053] Pooling layer: reduces the dimension of feature maps and reduces the amount of calculation;

[0054] Fully connected layer: expands the feature map and maps it to the output space;

[0055] Output layer: predict gesture category or motion trajectory;

[0056] (2) Convolutional layer formula:

[0057] Convolution operation: Z i,j,k =(W k *X) i,j +b k

[0058] Among them, Z i,j,k is the output of the kth convolution kernel at position (i, j), W k is the kth convolution kernel, X is the input, b k is the bias term, * is the convolution operation;

[0059] (3) Activation function:

[0060] A i,j,k =max(0,Zi,j,k )

[0061] (4) Pooling layer formula:

[0062] Max Pooling:

[0063] (5) Fully connected layer formula:

[0064] Weight matrix W fc and bias b fc :

[0065] F=W fc ·P+b fc

[0066] (6) Loss function:

[0067] Use the cross entropy loss function for classification tasks:

[0068]

[0069] Among them, y i It's really a label. is the predicted probability.

[0070] As a preferred solution of the mechanical arm force feedback control method based on deep reinforcement learning described in the present invention, the prediction model of the mechanical arm motion trajectory in step three includes:

[0071] The model architecture involves:

[0072] Input layer: input the current state and environmental information of the robot arm, including position, speed and acceleration;

[0073] Hidden layer: LSTM (Long Short-Term Memory Network) is used to capture long dependencies in time series;

[0074] Output layer: predict the future position and trajectory of the robotic arm;

[0075] LSTM formula:

[0076] Input gate: i t =σ(W i ·[h t-1 ,x t ]+b i )

[0077] Forget gate: f t =σ(W f ·[h t-1 ,x t ]+b f )

[0078] Memory Update:

[0079] Unit Status:

[0080] Output gate: o t =σ(W o ·[h t-1 ,x t ]+b o )

[0081] Hidden state: h t =o t tanh(C t )

[0082] Loss function:

[0083] Use mean squared error loss function for regression tasks:

[0084]

[0085] Among them, y i is the true value, is the predicted value, and N is the number of samples.

[0086] As a preferred solution of the mechanical arm force feedback control method based on deep reinforcement learning described in the present invention, the simulation test process in step 4 includes:

[0087] Data preparation:

[0088] Collect a large amount of gesture and robotic arm motion data, perform data preprocessing, and split it into training sets, validation sets, and test sets;

[0089] Model training:

[0090] Use the optimizer to perform backpropagation and gradient descent to update model parameters;

[0091] Perform cross-validation and adjust hyperparameters during training;

[0092] Model Validation and Testing:

[0093] Use the validation set to evaluate the prediction accuracy of the model and adjust the model structure and hyperparameters;

[0094] The generalization ability of the model is tested on the test set to ensure its robustness in practical applications.

[0095] As a preferred solution of the mechanical arm force feedback control method based on deep reinforcement learning described in the present invention, the specific process of using the convolutional neural network for gesture recognition during the simulation test includes:

[0096] Simulation data: Generate simulated gesture image data, which is divided into training set and test set;

[0097] Network architecture: including input layer, convolution layer, pooling layer, fully connected layer and output layer;

[0098] Training: Use cross entropy loss function and Adam optimizer for training;

[0099] Evaluation: Evaluate the accuracy of the model on the test set;

[0100] The specific process of robot arm motion trajectory prediction includes:

[0101] Simulation data: Generate the motion trajectory data of the robot arm, including position, velocity and acceleration;

[0102] Network architecture: LSTM model, including input layer, LSTM layer and output layer;

[0103] Training: Use mean square error loss function and Adam optimizer for training;

[0104] Evaluation: Evaluate the prediction error of the model on the test set.

[0105] The beneficial effects of the present invention are as follows: The present invention adopts adaptive deep reinforcement learning, and realizes dynamic adjustment and online update of the robot arm control strategy by designing an adaptive learning module, thereby improving its adaptability in complex environments; introduces a deep learning prediction model to predict human operation intentions and motion trajectories, optimizes the force feedback control strategy of the robot arm, and enhances the safety and efficiency of human-machine collaboration; proposes an overall control system architecture that integrates deep reinforcement learning, adaptive learning and prediction models to ensure efficient collaboration of each module and improve the overall performance of the robot arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0106] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0107] Figure 1 Schematic diagram of the method steps of the robot arm force feedback control method based on deep reinforcement learning of the present invention.

[0108] Figure 2 It is a line graph of the training accuracy and verification accuracy of the robot arm force feedback control method based on deep reinforcement learning of the present invention.

[0109] Figure 3It is a line graph of training loss and verification loss of the robot arm force feedback control method based on deep reinforcement learning of the present invention.

[0110] Figure 4 It is a line graph of the actual value and predicted value trajectory of the robot arm force feedback control method based on deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0111] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0112] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0113] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0114] Secondly, the present invention is described in detail with reference to the schematic diagram. When describing the embodiments of the present invention in detail, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0115] Reference Figure 1 - Figure 4 , provides a mechanical arm force feedback control method based on deep reinforcement learning, comprising the following steps:

[0116] Step 1: Data collection and preprocessing

[0117] Environmental data collection: Various data about the robot's surroundings, such as temperature, humidity, light, and object position, are collected in real time through a sensor network;

[0118] Operation data collection: collect motion data of human operators, including gestures, force feedback, operation trajectory and intention data;

[0119] Data preprocessing: De-noising, normalization and feature extraction of the collected data to form feature vectors for training deep reinforcement learning models;

[0120] Step 2: Adaptive Deep Reinforcement Learning Algorithm Design

[0121] Initial model training: Using the preprocessed data set, the deep reinforcement learning algorithm is used to train the initial control strategy of the robot arm;

[0122] Adaptive learning module: Design an adaptive learning module that can dynamically adjust the parameters of the deep reinforcement learning model and update the control strategy in real time according to changes in the real-time environment and operation data;

[0123] Model online update: Implement the online update function of the model, so that the robot arm can adapt to environmental changes and unknown interference through continuous learning and optimization during operation;

[0124] Step 3: Introduction of prediction model

[0125] Prediction of human operation intention: Use the convolutional neural network deep learning model to predict the gestures and movement trajectories of human operators and determine their operation intentions;

[0126] Motion trajectory prediction: Based on the collected human operation data, a prediction model is established to predict the motion trajectory of the robot arm, thereby optimizing its control strategy;

[0127] Step 4: Performance testing and optimization

[0128] Simulation testing: Test the performance of the control algorithm in a virtual simulation environment to verify its adaptability and stability in different scenarios and environments;

[0129] Safety and efficiency assessment: Evaluate the safety and operational efficiency of the robot arm in human-machine collaboration to ensure that it meets actual application requirements.

[0130] The initial model training in step 2 uses a deep Q network as a deep reinforcement learning algorithm, and the specific formula is as follows:

[0131] Q function update:

[0132]

[0133] Among them, s represents the current state, a represents the current action, r represents the immediate reward, s′ represents the next state, α represents the learning rate, and γ represents the discount factor;

[0134] Use a neural network to approximate the Q function:

[0135] Q(s,a;θ)≈Q(s,a)

[0136] Among them, θ is the parameter of the neural network;

[0137] Loss function:

[0138]

[0139] Among them, θ - is the parameter of the objective function, and D is the sample set in the experience replay pool;

[0140] Parameter update:

[0141]

[0142] Target network parameter update:

[0143] θ - ←θ(periodic update).

[0144] The adaptive learning module in step 2 is used to dynamically adjust the parameters of the deep reinforcement learning model to achieve real-time update control, including:

[0145] Real-time data collection: collect new samples from environment and operation data in real time and add them to the experience replay pool D;

[0146] Dynamically adjust the learning rate: Use the adaptive learning rate algorithm to adjust the learning rate of the neural network:

[0147]

[0148] Among them, α t is the learning rate at the tth iteration. The adaptive algorithm adaptively adjusts the learning rate based on the first-order moment estimate and the second-order moment estimate of the gradient:

[0149] m t =β1m t-1 +(1-β1)g t

[0150]

[0151] Among them, g t is the current gradient, β1 and β2 are exponential decay rates, ∈ is a decimal to prevent division by zero; m t is the first-order moment estimate of the gradient; v t is the second moment estimate of the gradient; is the bias correction of the first-order moment estimate; is the bias correction value of the second-order moment estimate; t is the current time step.

[0152] Furthermore, the step 2 also includes experience replay and priority sampling process, introducing a priority experience replay mechanism to prioritize samples with larger sampling errors:

[0153]

[0154] Among them, P(i) is the probability of sample i being sampled, δ iis the TD error of sample i, w is the hyperparameter for adjusting the sampling bias;

[0155] The model is updated online by combining newly collected data with the samples from the experience replay pool with priority sampling, and training is performed regularly to update the model parameters θ:

[0156]

[0157] The convolutional neural network used for gesture and motion trajectory prediction in step 3 includes:

[0158] (1) Network architecture design:

[0159] Input layer: input gesture and motion trajectory data, usually image sequence or time series;

[0160] Convolutional layer: extract spatial features;

[0161] Pooling layer: reduces the dimension of feature maps and reduces the amount of calculation;

[0162] Fully connected layer: expands the feature map and maps it to the output space;

[0163] Output layer: predict gesture category or motion trajectory;

[0164] (2) Convolutional layer formula:

[0165] Convolution operation: Z i,j,k =(W k *X) i,j +b k

[0166] Among them, Z i,j,k is the output of the kth convolution kernel at position (i, j), W k is the kth convolution kernel, X is the input, b k is the bias term, * is the convolution operation;

[0167] (3) Activation function:

[0168] A i,j,k =max(0,Z i,j,k )

[0169] (4) Pooling layer formula:

[0170] Max Pooling:

[0171] (5) Fully connected layer formula:

[0172] Weight matrix W fc and bias b fc :

[0173] F=Wfc ·P+b fc

[0174] (6) Loss function:

[0175] Use the cross entropy loss function for classification tasks:

[0176]

[0177] Among them, y i It's really a label. is the predicted probability.

[0178] Furthermore, the simulation test process in step 4 includes:

[0179] Data preparation:

[0180] Collect a large amount of gesture and robotic arm motion data, perform data preprocessing, and split it into training sets, validation sets, and test sets;

[0181] Model training:

[0182] Use the optimizer to perform backpropagation and gradient descent to update model parameters;

[0183] Perform cross-validation and adjust hyperparameters during training;

[0184] Model Validation and Testing:

[0185] Use the validation set to evaluate the prediction accuracy of the model and adjust the model structure and hyperparameters;

[0186] The generalization ability of the model is tested on the test set to ensure its robustness in practical applications.

[0187] Furthermore, the specific process of using the convolutional neural network for gesture recognition during the simulation test includes:

[0188] Simulation data: Generate simulated gesture image data, which is divided into training set and test set;

[0189] Network architecture: including input layer, convolution layer, pooling layer, fully connected layer and output layer;

[0190] Training: Use cross entropy loss function and Adam optimizer for training;

[0191] Evaluation: Evaluate the accuracy of the model on the test set;

[0192] The specific process of robot arm motion trajectory prediction includes:

[0193] Simulation data: Generate the motion trajectory data of the robot arm, including position, velocity and acceleration;

[0194] Network architecture: LSTM model, including input layer, LSTM layer and output layer;

[0195] Training: Use mean square error loss function and Adam optimizer for training;

[0196] Evaluation: Evaluate the prediction error of the model on the test set.

[0197] Reference Figure 2 -4:

[0198] Example 1: Convolutional Neural Network for Gesture Recognition

[0199] Simulated data: Generate simulated gesture image data, which is divided into training set and test set.

[0200] Network architecture: A simple CNN model, including input layer, convolution layer, pooling layer, fully connected layer and output layer.

[0201] Training: The cross entropy loss function and Adam optimizer are used for training.

[0202] Evaluation: Evaluate the accuracy of the model on the test set.

[0203] Example 2: LSTM for predicting robot arm motion trajectory

[0204] Simulation data: Generate the motion trajectory data of the robot arm, including position, velocity and acceleration.

[0205] Network architecture: A simple LSTM model, including input layer, LSTM layer and output layer.

[0206] Training: Training is performed using the mean squared error loss function and the Adam optimizer.

[0207] Evaluation: Evaluate the prediction error of the model on the test set.

[0208] Specific implementation process:

[0209] Example 1: Convolutional Neural Network (CNN) for Gesture Recognition

[0210] Simulated data generation:

[0211] Gesture data: Generate 1000 28x28 random grayscale images, each image corresponds to a gesture category (5 categories in total).

[0212] Network architecture:

[0213] Input layer: Input gesture image data, the shape is (28, 28, 1).

[0214] Convolutional layer: 32 3x3 convolution kernels, ReLU activation function.

[0215] Pooling layer: 2x2 max pooling.

[0216] Fully connected layer: 64 neurons, ReLU activation function.

[0217] Output layer: 5 neurons, softmax activation function.

[0218] train:

[0219] Optimizer: Adam.

[0220] Loss function: categorical_crossentropy.

[0221] Number of training rounds: 10.

[0222] Validation set ratio: 10%.

[0223] Evaluation: Evaluate the accuracy of the model on the test set.

[0224] Example 2: LSTM for predicting robot arm motion trajectory

[0225] Simulated data generation:

[0226] Robotic arm motion data: Generate 1000 samples, each containing 3 features (position, velocity, acceleration) for 10 time steps.

[0227] Network architecture:

[0228] Input layer: Input robot arm motion data, shape is (10, 3).

[0229] LSTM layer: 50 units, ReLU activation function.

[0230] Output layer: 3 neurons, used to predict 3 future feature values.

[0231] train:

[0232] Optimizer: Adam.

[0233] Loss function: Mean Squared Error (MSE).

[0234] Number of training rounds: 10.

[0235] Validation set ratio: 10%.

[0236] Evaluation: Evaluate the prediction error of the model on the test set.

[0237] Example 1: Gesture recognition data table:

[0238]

[0239] Example 2: Robotic arm motion trajectory data table

[0240]

[0241] Results

[0242] like Figure 2 As shown, Example 1: a line chart of training and validation accuracy shows the training process and accuracy of the model.

[0243] like Figure 3 and Figure 4 As shown in Example 2: a line graph of training and validation losses, and a comparison graph of actual values ​​and predicted values, showing the prediction effect of the model.

[0244] A computer device includes a memory and a processor. The computer device may be a terminal. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner may be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device may be a touch layer covering the display screen, or may be a key, trackball or touchpad provided on the housing of the computer device, or may be an external keyboard, touchpad or mouse, etc.

[0245] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, a method for implementing a mechanical arm force feedback control method based on deep reinforcement learning as proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, disk or optical disk.

[0246] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A mechanical arm force feedback control method based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Data collection and preprocessing Environmental data collection: Various data about the robot's surroundings, such as temperature, humidity, light, and object position, are collected in real time through a sensor network; Operation data collection: collect motion data of human operators, including gestures, force feedback, operation trajectory and intention data; Data preprocessing: De-noising, normalization and feature extraction of the collected data to form feature vectors for training deep reinforcement learning models; Step 2: Adaptive Deep Reinforcement Learning Algorithm Design Initial model training: Using the preprocessed data set, the deep reinforcement learning algorithm is used to train the initial control strategy of the robot arm; Adaptive learning module: Design an adaptive learning module that can dynamically adjust the parameters of the deep reinforcement learning model and update the control strategy in real time according to changes in the real-time environment and operation data; Model online update: Implement the online update function of the model, so that the robot arm can adapt to environmental changes and unknown interference through continuous learning and optimization during operation; Step 3: Introduction of prediction model Prediction of human operation intention: Use the convolutional neural network deep learning model to predict the gestures and movement trajectories of human operators and determine their operation intentions; Motion trajectory prediction: Based on the collected human operation data, a prediction model is established to predict the motion trajectory of the robot arm, thereby optimizing its control strategy; Step 4: Performance testing and optimization Simulation testing: Test the performance of the control algorithm in a virtual simulation environment to verify its adaptability and stability in different scenarios and environments; Safety and efficiency assessment: Evaluate the safety and operational efficiency of the robot arm in human-machine collaboration to ensure that it meets actual application requirements.

2. The mechanical arm force feedback control method based on deep reinforcement learning according to claim 1, characterized in that: The initial model training in step 2 uses a deep Q network as a deep reinforcement learning algorithm. The specific formula is as follows: Q function update: Among them, s represents the current state, a represents the current action, r represents the immediate reward, s′ represents the next state, α represents the learning rate, and γ represents the discount factor; Use a neural network to approximate the Q function: Q(s, a; θ)≈Q(s, a) Among them, θ is the parameter of the neural network; Loss function: Among them, θ - is the parameter of the objective function, and D is the sample set in the experience replay pool; Parameter update: Target network parameter update: θ - ←θ(periodic update).

3. The mechanical arm force feedback control method based on deep reinforcement learning according to claim 2, characterized in that: The adaptive learning module in step 2 is used to dynamically adjust the parameters of the deep reinforcement learning model to achieve real-time update control, including: Real-time data collection: collect new samples from environment and operation data in real time and add them to the experience replay pool D; Dynamically adjust the learning rate: Use the adaptive learning rate algorithm to adjust the learning rate of the neural network: Among them, α t is the learning rate at the tth iteration. The adaptive algorithm adaptively adjusts the learning rate based on the first-order moment estimate and the second-order moment estimate of the gradient: m t =β1m t-1 +(1-β1)g t Among them, g t is the current gradient, β1 and β2 are exponential decay rates, and ∈ is a decimal to prevent division by zero; m t is the first-order moment estimate of the gradient; v t is the second moment estimate of the gradient; is the bias correction of the first-order moment estimate; is the bias correction value of the second-order moment estimate; t is the current time step.

4. The mechanical arm force feedback control method based on deep reinforcement learning according to claim 3 is characterized in that: The step 2 also includes experience replay and priority sampling process, introducing a priority experience replay mechanism to give priority to samples with larger sampling errors: Among them, P(i) is the probability of sample i being sampled, δ i is the TD error of sample i, w is the hyperparameter for adjusting the sampling bias; The model is updated online by combining newly collected data with the samples from the experience replay pool with priority sampling, and training is performed regularly to update the model parameters θ:

5. The mechanical arm force feedback control method based on deep reinforcement learning according to claim 1, characterized in that: In step 3, the convolutional neural network is used for gesture and motion trajectory prediction, including: (1) Network architecture design: Input layer: input gesture and motion trajectory data, usually image sequence or time series; Convolutional layer: extract spatial features; Pooling layer: reduces the dimension of feature maps and reduces the amount of calculation; Fully connected layer: expands the feature map and maps it to the output space; Output layer: predict gesture category or motion trajectory; (2) Convolutional layer formula: Convolution operation: Z i，j，k =(W k *X) i，j +b k Among them, Z i，j，k is the output of the kth convolution kernel at position (i, j), W k is the kth convolution kernel, X is the input, b k is the bias term, * is the convolution operation; (3) Activation function: A i，j，k =max(0,Z i，j，k ) (4) Pooling layer formula: Max Pooling: (5) Fully connected layer formula: Weight matrix W fc and bias b fc : F=W fc ·P+b fc (6) Loss function: Use the cross entropy loss function for classification tasks: Among them, yi is the real label, is the predicted probability.

6. The mechanical arm force feedback control method based on deep reinforcement learning according to claim 5, characterized in that: The prediction model of the robot arm motion trajectory in step 3 includes: The model architecture involves: Input layer: input the current state and environmental information of the robot arm, including position, speed and acceleration; Hidden layer: LSTM (Long Short-Term Memory Network) is used to capture long dependencies in time series; Output layer: predict the future position and trajectory of the robotic arm; LSTM formula: Input gate: i t =σ(W v ·[h t-1 , x t ]+b i ) Forget gate: f t =σ(W f ·[h t-1 , x t ]+b f ) Memory Update: Unit Status: Output gate: o t =σ(W o ·[h t-1 , x t ]+b o ) Hidden state: h t =o t tanh(C t ) Loss function: Use mean squared error loss function for regression tasks: Among them, y i is the true value, is the predicted value, and N is the number of samples.

7. The mechanical arm force feedback control method based on deep reinforcement learning according to claim 6, characterized in that: The simulation test process in step 4 includes: Data preparation: Collect a large amount of gesture and robotic arm motion data, perform data preprocessing, and split it into training sets, validation sets, and test sets; Model training: Use the optimizer to perform backpropagation and gradient descent to update model parameters; Perform cross-validation and adjust hyperparameters during training; Model Validation and Testing: Use the validation set to evaluate the prediction accuracy of the model and adjust the model structure and hyperparameters; The generalization ability of the model is tested on the test set to ensure its robustness in practical applications.

8. The mechanical arm force feedback control method based on deep reinforcement learning according to claim 7, characterized in that: The specific process of using the convolutional neural network for gesture recognition during the simulation test includes: Simulation data: Generate simulated gesture image data, which is divided into training set and test set; Network architecture: including input layer, convolution layer, pooling layer, fully connected layer and output layer; Training: Use cross entropy loss function and Adam optimizer for training; Evaluation: Evaluate the accuracy of the model on the test set; The specific process of robot arm motion trajectory prediction includes: Simulation data: Generate the motion trajectory data of the robot arm, including position, velocity and acceleration; Network architecture: LSTM model, including input layer, LSTM layer and output layer; Training: Use mean square error loss function and Adam optimizer for training; Evaluation: Evaluate the prediction error of the model on the test set.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of a robot arm force feedback control method based on deep reinforcement learning as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a robot arm force feedback control method based on deep reinforcement learning as described in any one of claims 1 to 8 are implemented.

Images