A mechanical arm force feedback control method based on deep reinforcement learning

Abstract

The application discloses a kind of mechanical arm force feedback control methods based on deep reinforcement learning, comprising the following steps: step one: data acquisition and pretreatment, including environmental data acquisition, operation data acquisition and data pretreatment;Step two: adaptive deep reinforcement learning algorithm design, including initial model training, adaptive learning module and model online update;Step three: prediction model is introduced, including human operation intention prediction and motion trajectory prediction;Step four: performance test and optimization, including simulation test and safety and efficiency evaluation.The application improves the force feedback control precision of mechanical arm in complex environment and the safety and efficiency of man-machine cooperation.This method realizes real-time update and adjustment of control strategy by designing adaptive deep reinforcement learning algorithm and introducing human operation intention prediction model, so as to better cope with environmental changes and unknown disturbances.

Description

Technical Field

[0001] This invention relates to the technical field of robotic arm control, and in particular to a force feedback control method for robotic arms based on deep reinforcement learning. Background Technology

[0002] Existing robotic arm control processes typically combine multiple sensors (such as force sensors, position sensors, vision sensors, etc.) to sense the environment and provide force feedback.

[0003] However, when robotic arms operate in complex environments, force feedback control methods need to adapt quickly to changes and disturbances, and traditional control methods lack flexibility. In addition, ensuring safety and efficiency when robotic arms collaborate with humans is a challenge, and existing methods struggle to achieve both efficiency and safety. To address these issues, we propose a deep reinforcement learning-based force feedback control method for robotic arms. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] In view of the problems existing in the current deep reinforcement learning-based force feedback control methods for robotic arms, this invention is proposed.

[0006] Therefore, the purpose of this invention is to provide a force feedback control method for robotic arms based on deep reinforcement learning, which improves the force feedback control accuracy and the safety and efficiency of human-robot collaboration when the robotic arm operates in complex environments. This method achieves real-time updates and adjustments to the control strategy by designing an adaptive deep reinforcement learning algorithm and introducing a human operational intent prediction model, thereby better coping with environmental changes and unknown disturbances.

[0007] To address the aforementioned technical problems, this invention provides the following technical solution: a force feedback control method for a robotic arm based on deep reinforcement learning, comprising the following steps:

[0008] Step 1: Data Acquisition and Preprocessing

[0009] Environmental data acquisition: Real-time acquisition of various data about the environment around the robotic arm, such as temperature, humidity, light, and object position, through a sensor network;

[0010] Operational data acquisition: Collecting motion data of human operators, including gestures, force feedback, operation trajectory, and intention data;

[0011] Data preprocessing: The collected data is denoised, normalized, and feature extracted to form feature vectors for training deep reinforcement learning models;

[0012] Step 2: Adaptive Deep Reinforcement Learning Algorithm Design

[0013] Initial model training: Using the preprocessed dataset, a deep reinforcement learning algorithm is used to train the initial control strategy of the robotic arm;

[0014] Adaptive learning module: Design an adaptive learning module that can dynamically adjust the parameters of the deep reinforcement learning model and update the control strategy in real time according to changes in the real-time environment and operational data.

[0015] Online model update: Enables online model updates, allowing the robotic arm to adapt to environmental changes and unknown interferences through continuous learning and optimization during operation;

[0016] Step 3: Introduction of the Prediction Model

[0017] Human operational intent prediction: Using a convolutional neural network deep learning model, the gestures and movement trajectories of human operators are predicted to determine their operational intent.

[0018] Motion trajectory prediction: Based on collected human operation data, a predictive model is built to predict the motion trajectory of the robotic arm, thereby optimizing its control strategy;

[0019] Step 4: Performance Testing and Optimization

[0020] Simulation testing: Test the performance of the control algorithm in a virtual simulation environment to verify its adaptability and stability in different scenarios and environments;

[0021] Safety and efficiency assessment: Evaluate the safety and operational efficiency of the robotic arm in human-robot collaboration to ensure it meets the needs of practical applications.

[0022] As a preferred embodiment of the deep reinforcement learning-based robotic arm force feedback control method of the present invention, wherein: the initial model training in step two uses a deep Q-network as a deep reinforcement learning algorithm, and the specific formula is as follows:

[0023] Q function update:

[0024]

[0025] Where s represents the current state, a represents the current action, r represents the immediate reward, s′ represents the next state, α represents the learning rate, and γ represents the discount factor;

[0026] Approximating the Q function using a neural network:

[0027] Q(s,a;θ)≈Q(s,a)

[0028] Where θ are the parameters of the neural network;

[0029] Loss function:

[0030]

[0031] Where, θ - is the parameter of the objective function, and D is the sample set in the experience replay pool;

[0032] Parameter update:

[0033]

[0034] Target network parameter update:

[0035] θ - ←θ (Periodic update).

[0036] As a preferred embodiment of the deep reinforcement learning-based robotic arm force feedback control method of the present invention, wherein: the adaptive learning module in step two is used to dynamically adjust the parameters of the deep reinforcement learning model to achieve real-time update control, including:

[0037] Real-time data acquisition: New samples are acquired in real time from environmental and operational data and added to the experience replay pool D;

[0038] Dynamically adjusting the learning rate: Using an adaptive learning rate algorithm to adjust the learning rate of the neural network.

[0039]

[0040] Where, α t This is the learning rate at the t-th iteration. The adaptive algorithm adaptively adjusts the learning rate based on the first and second moment estimates of the gradient.

[0041] m t =β1m t-1 +(1-β1)g t

[0042]

[0043] Among them, g t This is the current gradient, β1 and β2 are the exponential decay rates, and ∈ is a decimal to prevent division by zero; m t It is the first moment estimate of the gradient; v t It is the second moment estimate of the gradient; It is the bias correction value for the first moment estimate; t is the bias correction value for the second-order moment estimation; t is the current time step.

[0044] As a preferred embodiment of the deep reinforcement learning-based robotic arm force feedback control method described in this invention, step two further includes an experience playback and priority sampling process, introducing a priority experience playback mechanism to prioritize sampling samples with larger errors.

[0045]

[0046] Where P(i) is the probability that sample i is sampled, δ i is the TD error of sample i, and w is the hyperparameter for adjusting the sampling bias;

[0047] The model is updated online by combining newly acquired data and preferentially sampled empirical replay pool samples, and is trained periodically to update the model's parameters θ.

[0048]

[0049] As a preferred embodiment of the deep reinforcement learning-based robotic arm force feedback control method of the present invention, wherein: the convolutional neural network used for gesture and motion trajectory prediction in step three includes:

[0050] (1) Network architecture design:

[0051] Input layer: Input gestures and motion trajectory data, usually image sequences or time series;

[0052] Convolutional layers: extract spatial features;

[0053] Pooling layers: reduce the dimensionality of feature maps and decrease computational cost;

[0054] Fully connected layer: Unfolds the feature map and maps it to the output space;

[0055] Output layer: Predicts gesture category or motion trajectory;

[0056] (2) Convolutional layer formula:

[0057] Convolution operation: Z i,j,k =(W k *X) i,j +b k

[0058] Among them, Z i,j,k W is the output of the k-th convolutional kernel at position (i,j). k It is the k-th convolutional kernel, X is the input, b k is the bias term, and * is the convolution operation;

[0059] (3) Activation function:

[0060] A i,j,k =max(0,Z)i,j,k )

[0061] (4) Pooling layer formula:

[0062] Max pooling:

[0063] (5) Fully connected layer formula:

[0064] Weight matrix W fc and bias b fc :

[0065] F = W fc ·P+b fc

[0066] (6) Loss function:

[0067] Using the cross-entropy loss function for classification tasks:

[0068]

[0069] Among them, y i It's a genuine label. It is a predicted probability.

[0070] As a preferred embodiment of the deep reinforcement learning-based robotic arm force feedback control method of the present invention, wherein the prediction model of the robotic arm motion trajectory in step three includes:

[0071] The model architecture involves:

[0072] Input layer: Inputs the current state and environmental information of the robotic arm, including position, velocity, and acceleration;

[0073] Hidden layers: LSTM (Long Short-Term Memory) networks are used to capture long dependencies in time series data;

[0074] Output layer: Predicts the future position and trajectory of the robotic arm;

[0075] LSTM formula:

[0076] Input gate: i t =σ(W i ·[h t-1 ,x t ]+b i )

[0077] Forgotten Gate: f t =σ(W f ·[h t-1 ,x t ]+b f )

[0078] Memory update:

[0079] Cell status:

[0080] Output gate: o t =σ(W o ·[h t-1 ,x t ]+b o )

[0081] Hidden state: h t =o t ·tanh(C t )

[0082] Loss function:

[0083] Using the mean squared error loss function for regression tasks:

[0084]

[0085] Among them, y i It is the actual value. is the predicted value, and N is the number of samples.

[0086] As a preferred embodiment of the deep reinforcement learning-based force feedback control method for robotic arms described in this invention, the simulation testing process in step four includes:

[0087] Data preparation:

[0088] A large amount of gesture and robotic arm motion data was collected, preprocessed, and divided into training, validation, and test sets.

[0089] Model training:

[0090] Use the optimizer to perform backpropagation and gradient descent to update the model parameters;

[0091] Cross-validation is performed during training, and hyperparameters are adjusted.

[0092] Model validation and testing:

[0093] Use the validation set to evaluate the model's prediction accuracy and adjust the model structure and hyperparameters.

[0094] Test the model's generalization ability on the test set to ensure its robustness in real-world applications.

[0095] As a preferred embodiment of the deep reinforcement learning-based robotic arm force feedback control method of the present invention, the specific process of using the convolutional neural network for gesture recognition during the simulation test includes:

[0096] Simulated data: Generating simulated gesture image data, divided into training set and test set;

[0097] Network architecture includes an input layer, convolutional layers, pooling layers, fully connected layers, and an output layer;

[0098] Training: Training was performed using the cross-entropy loss function and the Adam optimizer;

[0099] Evaluation: Evaluate the model's accuracy on the test set;

[0100] The specific process of predicting the motion trajectory of a robotic arm includes:

[0101] Simulated data: Generates motion trajectory data for the robotic arm, including position, velocity, and acceleration;

[0102] Network architecture: LSTM model, including input layer, LSTM layer and output layer;

[0103] Training: Training is performed using the mean squared error loss function and the Adam optimizer;

[0104] Evaluation: Evaluate the model's prediction error on the test set.

[0105] The beneficial effects of this invention are as follows: This invention employs adaptive deep reinforcement learning, and through the design of an adaptive learning module, it realizes the dynamic adjustment and online updating of the robotic arm's control strategy, thereby improving its adaptability in complex environments; it introduces a deep learning prediction model to predict human operating intentions and motion trajectories, optimizes the robotic arm's force feedback control strategy, and enhances the safety and efficiency of human-machine collaboration; it proposes an overall control system architecture that integrates deep reinforcement learning, adaptive learning, and prediction models, ensuring the efficient collaborative work of each module and improving the overall performance of the robotic arm. Attached Figure Description

[0106] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0107] Figure 1 This is a schematic diagram illustrating the method steps of the robotic arm force feedback control method based on deep reinforcement learning according to the present invention.

[0108] Figure 2 This is a line graph showing the training accuracy and verification accuracy of the deep reinforcement learning-based force feedback control method for robotic arms according to the present invention.

[0109] Figure 3This is a line graph showing the training loss and validation loss of the deep reinforcement learning-based force feedback control method for robotic arms according to the present invention.

[0110] Figure 4 This is a line graph showing the actual and predicted values ​​of the force feedback control method for robotic arms based on deep reinforcement learning, as described in this invention. Detailed Implementation

[0111] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0112] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0113] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0114] Secondly, the present invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of the present invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not according to the usual scale. Furthermore, the schematic diagrams are merely examples and should not limit the scope of protection of the present invention. In addition, actual fabrication should include three-dimensional spatial dimensions of length, width, and depth.

[0115] Reference Figure 1 - Figure 4 This paper presents a force feedback control method for a robotic arm based on deep reinforcement learning, which includes the following steps:

[0116] Step 1: Data Acquisition and Preprocessing

[0117] Environmental data acquisition: Real-time acquisition of various data about the environment around the robotic arm, such as temperature, humidity, light, and object position, through a sensor network;

[0118] Operational data acquisition: Collecting motion data of human operators, including gestures, force feedback, operation trajectory, and intention data;

[0119] Data preprocessing: The collected data is denoised, normalized, and feature extracted to form feature vectors for training deep reinforcement learning models;

[0120] Step 2: Adaptive Deep Reinforcement Learning Algorithm Design

[0121] Initial model training: Using the preprocessed dataset, a deep reinforcement learning algorithm is used to train the initial control strategy of the robotic arm;

[0122] Adaptive learning module: Design an adaptive learning module that can dynamically adjust the parameters of the deep reinforcement learning model and update the control strategy in real time according to changes in the real-time environment and operational data.

[0123] Online model update: Enables online model updates, allowing the robotic arm to adapt to environmental changes and unknown interferences through continuous learning and optimization during operation;

[0124] Step 3: Introduction of the Prediction Model

[0125] Human operational intent prediction: Using a convolutional neural network deep learning model, the gestures and movement trajectories of human operators are predicted to determine their operational intent.

[0126] Motion trajectory prediction: Based on collected human operation data, a predictive model is built to predict the motion trajectory of the robotic arm, thereby optimizing its control strategy;

[0127] Step 4: Performance Testing and Optimization

[0128] Simulation testing: Test the performance of the control algorithm in a virtual simulation environment to verify its adaptability and stability in different scenarios and environments;

[0129] Safety and efficiency assessment: Evaluate the safety and operational efficiency of the robotic arm in human-robot collaboration to ensure it meets the needs of practical applications.

[0130] In step two, the initial model training uses a deep Q-network as a deep reinforcement learning algorithm, and the specific formula is as follows:

[0131] Q function update:

[0132]

[0133] Where s represents the current state, a represents the current action, r represents the immediate reward, s′ represents the next state, α represents the learning rate, and γ represents the discount factor;

[0134] Approximating the Q function using a neural network:

[0135] Q(s,a;θ)≈Q(s,a)

[0136] Where θ are the parameters of the neural network;

[0137] Loss function:

[0138]

[0139] Where, θ - is the parameter of the objective function, and D is the sample set in the experience replay pool;

[0140] Parameter update:

[0141]

[0142] Target network parameter update:

[0143] θ - ←θ (Periodic update).

[0144] In step two, the adaptive learning module is used to dynamically adjust the parameters of the deep reinforcement learning model to achieve real-time update control, including:

[0145] Real-time data acquisition: New samples are acquired in real time from environmental and operational data and added to the experience replay pool D;

[0146] Dynamically adjusting the learning rate: Using an adaptive learning rate algorithm to adjust the learning rate of the neural network.

[0147]

[0148] Where, α t This is the learning rate at the t-th iteration. The adaptive algorithm adaptively adjusts the learning rate based on the first and second moment estimates of the gradient.

[0149] m t =β1m t-1 +(1-β1)g t

[0150]

[0151] Among them, g t This is the current gradient, β1 and β2 are the exponential decay rates, and ∈ is a decimal to prevent division by zero; m t It is the first moment estimate of the gradient; v t It is the second moment estimate of the gradient; It is the bias correction value for the first moment estimate; t is the bias correction value for the second-order moment estimation; t is the current time step.

[0152] Furthermore, step two also includes an experience replay and priority sampling process, introducing a priority experience replay mechanism to prioritize sampling samples with larger errors:

[0153]

[0154] Where P(i) is the probability that sample i is sampled, δ iis the TD error of sample i, and w is the hyperparameter for adjusting the sampling bias;

[0155] The model is updated online by combining newly acquired data and preferentially sampled empirical replay pool samples, and is trained periodically to update the model's parameters θ.

[0156]

[0157] In step three, the convolutional neural network used for gesture and motion trajectory prediction includes:

[0158] (1) Network architecture design:

[0159] Input layer: Input gestures and motion trajectory data, usually image sequences or time series;

[0160] Convolutional layers: extract spatial features;

[0161] Pooling layers: reduce the dimensionality of feature maps and decrease computational cost;

[0162] Fully connected layer: Unfolds the feature map and maps it to the output space;

[0163] Output layer: Predicts gesture category or motion trajectory;

[0164] (2) Convolutional layer formula:

[0165] Convolution operation: Z i,j,k =(W k *X) i,j +b k

[0166] Among them, Z i,j,k W is the output of the k-th convolutional kernel at position (i,j). k It is the k-th convolutional kernel, X is the input, b k is the bias term, and * is the convolution operation;

[0167] (3) Activation function:

[0168] A i,j,k =max(0,Z) i,j,k )

[0169] (4) Pooling layer formula:

[0170] Max pooling:

[0171] (5) Fully connected layer formula:

[0172] Weight matrix W fc and bias b fc :

[0173] F = Wfc ·P+b fc

[0174] (6) Loss function:

[0175] Using the cross-entropy loss function for classification tasks:

[0176]

[0177] Among them, y i It's a genuine label. It is a predicted probability.

[0178] Furthermore, the simulation testing process in step four includes:

[0179] Data preparation:

[0180] A large amount of gesture and robotic arm motion data was collected, preprocessed, and divided into training, validation, and test sets.

[0181] Model training:

[0182] Use the optimizer to perform backpropagation and gradient descent to update the model parameters;

[0183] Cross-validation is performed during training, and hyperparameters are adjusted.

[0184] Model validation and testing:

[0185] Use the validation set to evaluate the model's prediction accuracy and adjust the model structure and hyperparameters.

[0186] Test the model's generalization ability on the test set to ensure its robustness in real-world applications.

[0187] Furthermore, the specific process of using the convolutional neural network for gesture recognition during the simulation test includes:

[0188] Simulated data: Generating simulated gesture image data, divided into training set and test set;

[0189] Network architecture includes an input layer, convolutional layers, pooling layers, fully connected layers, and an output layer;

[0190] Training: Training was performed using the cross-entropy loss function and the Adam optimizer;

[0191] Evaluation: Evaluate the model's accuracy on the test set;

[0192] The specific process of predicting the motion trajectory of a robotic arm includes:

[0193] Simulated data: Generates motion trajectory data for the robotic arm, including position, velocity, and acceleration;

[0194] Network architecture: LSTM model, including input layer, LSTM layer and output layer;

[0195] Training: Training is performed using the mean squared error loss function and the Adam optimizer;

[0196] Evaluation: Evaluate the model's prediction error on the test set.

[0197] Reference Figure 2 -4:

[0198] Example 1: Convolutional Neural Networks for Gesture Recognition

[0199] Simulated data: Generating simulated gesture image data, divided into training set and test set.

[0200] Network architecture: A simple CNN model, consisting of an input layer, convolutional layers, pooling layers, fully connected layers, and an output layer.

[0201] Training: Training was performed using the cross-entropy loss function and the Adam optimizer.

[0202] Evaluation: Evaluate the model's accuracy on the test set.

[0203] Example 2: LSTM for predicting the motion trajectory of a robotic arm

[0204] Simulation data: Generates motion trajectory data for the robotic arm, including position, velocity, and acceleration.

[0205] Network architecture: A simple LSTM model, consisting of an input layer, an LSTM layer, and an output layer.

[0206] Training: Training is performed using the mean squared error loss function and the Adam optimizer.

[0207] Evaluation: Evaluate the model's prediction error on the test set.

[0208] Specific implementation process:

[0209] Example 1: Convolutional Neural Network (CNN) for Gesture Recognition

[0210] Simulated data generation:

[0211] Gesture data: Generate 1000 random grayscale images of 28x28 pixels, each image corresponding to a gesture category (5 categories in total).

[0212] Network architecture:

[0213] Input layer: Input gesture image data, with a shape of (28, 28, 1).

[0214] Convolutional layer: 32 3x3 convolutional kernels, ReLU activation function.

[0215] Pooling layer: 2x2 max pooling.

[0216] Fully connected layer: 64 neurons, ReLU activation function.

[0217] Output layer: 5 neurons, softmax activation function.

[0218] train:

[0219] Optimizer: Adam.

[0220] Loss function: categorical_crossentropy.

[0221] Number of training rounds: 10.

[0222] Validation set ratio: 10%.

[0223] Evaluation: Evaluate the model's accuracy on the test set.

[0224] Example 2: LSTM for predicting the motion trajectory of a robotic arm

[0225] Simulated data generation:

[0226] Robotic arm motion data: 1000 samples are generated, each sample contains 3 features (position, velocity, acceleration) at 10 time steps.

[0227] Network architecture:

[0228] Input layer: Input the robotic arm motion data, with a shape of (10, 3).

[0229] LSTM layer: 50 units, ReLU activation function.

[0230] Output layer: 3 neurons, used to predict 3 future feature values.

[0231] train:

[0232] Optimizer: Adam.

[0233] Loss function: Mean Squared Error (MSE).

[0234] Number of training rounds: 10.

[0235] Validation set ratio: 10%.

[0236] Evaluation: Evaluate the model's prediction error on the test set.

[0237] Example 1: Gesture Recognition Data Table:

[0238]

[0239] Example 2: Robotic Arm Motion Trajectory Data Table

[0240]

[0241] Results Display

[0242] like Figure 2 As shown in Example 1: Line graph of training and validation accuracy, which shows the training process and accuracy of the model.

[0243] like Figure 3 and Figure 4 As shown in Example 2: Line graphs of training and validation losses, and comparison graphs of actual and predicted values, demonstrating the model's prediction performance.

[0244] A computer device includes a memory and a processor. This computer device can be a terminal. The computer device includes a processor, memory, a communication interface, a display screen, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad located on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0245] This embodiment also provides a storage medium on which a computer program is stored. When executed by a processor, the program implements a deep reinforcement learning-based force feedback control method for a robotic arm as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0246] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for force feedback control of a robotic arm based on deep reinforcement learning, the method comprising: Comprising the following steps: Step one: data collection and preprocessing Environmental data collection: Real-time collection of various data around the robot arm through sensor networks, including temperature, humidity, light, object position and speed; Operation data collection: Collect human operator's motion data, including gestures, force feedback, operation trajectory and intention data; Data preprocessing: Denoising, normalization and feature extraction of collected data to form feature vectors for training deep reinforcement learning model; Step two: adaptive deep reinforcement learning algorithm design Initial model training: Using preprocessed data set, using deep reinforcement learning algorithm to train the initial control strategy of the robot arm; Adaptive learning module: Design an adaptive module that can dynamically adjust the parameters of the deep reinforcement learning model according to the changes of real-time environment and operation data, and update the control strategy in real time; Online model update: Realize the online update function of the model, so that the robot arm can adapt to environmental changes and unknown disturbances by continuous learning and optimization during operation; Wherein, the initial model training uses deep Q network as deep reinforcement learning algorithm, the specific formula is as follows: Q function update: wherein, represents the current state, a represents the current action, r represents the immediate reward, represents the next state, represents the learning rate, represents the discount factor; represents the maximization operation, evaluating which of the next state actions yields the maximum Q value; Using neural network to approximate Q function: wherein, are parameters of the neural network; Loss function: wherein, is a parameter of the target function, D is a set of sample in the experience replay pool; denotes an expected value; is a transition sample tuple; Parameter update: Target network parameter update: periodically updated; denotes the gradient of the loss function with respect to the parameters ; Wherein, the adaptive module is used to dynamically adjust the parameters of the deep reinforcement learning model to realize real-time update control, including: Real-time data collection: Collect new samples from environmental and operation data and add them to experience replay pool D; Dynamic adjustment of learning rate: Adjust the learning rate of neural network using adaptive learning rate algorithm: wherein, is the learning rate at the tth iteration, which is adaptively adjusted by the adaptive algorithm according to the first and second moment estimates of the gradient: wherein, is the current gradient, , is the exponential decay rate, is a small number to prevent division by zero; is the first moment estimate of the gradient; is the second moment estimate of the gradient; is the bias correction value for the first moment estimate; is the bias correction value for the second moment estimate; is the current time step; In step two, experience replay and priority sampling process are also included, and priority experience replay mechanism is introduced to preferentially sample samples with larger error: wherein, is the probability that sample i was sampled, is the TD error for sample i, is a hyperparameter that regulates the bias of sampling; Real-time model updating: periodically train with newly collected data and prioritized samples from the experience replay pool to update the model's parameters : ； Step three: prediction model introduction Human operation intention prediction: Use convolutional neural network deep learning model to predict human operator's gestures and motion trajectory to determine their operation intention; Motion trajectory prediction: Based on the collected human operation data, establish a prediction model to predict the motion trajectory of the robot arm, so as to optimize its control strategy; Step four: performance test and optimization Simulation test: Test the performance of the control algorithm in virtual simulation environment to verify its adaptability and stability in different scenes and environments; Safety and efficiency evaluation: Evaluate the safety and operation efficiency of the robot arm in human-robot collaboration to ensure that it meets the actual application requirements.

2. The deep reinforcement learning based manipulator force feedback control method of claim 1, wherein: The convolutional neural network in step three is used for gesture and motion trajectory prediction, including: (1) Network architecture design: Input layer: input gesture and motion trajectory data, usually image sequence or time sequence; Convolution layer: extract spatial features; Pooling layer: reduce the dimension of feature map and reduce the amount of calculation; Fully connected layer: expand and map the feature map to the output space; Output layer: predict gesture category or motion trajectory; (2) Convolution layer formula: Convolution operation: wherein, is the output of the k-th convolution kernel at position , is the k-th convolution kernel, is the input, is the bias term, is the convolution operation; (3) Activation function: denotes the activated feature value; (4) Pooling layer formula: max pooling: represents the pooled feature values; (5) Fully connected layer formula: weight matrix and bias : denotes the weight matrix of the fully connected layer; denotes the bias term for the fully connected layer; denotes a feature vector representing the output of the pooling layer; (6) Loss function: Use cross entropy loss function for classification task: wherein, is a true label, is a predicted probability.

3. The deep reinforcement learning based manipulator force feedback control method of claim 2, wherein: The prediction model of the robot arm motion trajectory in step three includes: Model architecture involves: Input layer: input the current state and environmental information of the robot arm, including position, velocity, and acceleration. Hidden layer: long short-term memory network is used to capture long dependencies in time series. Output layer: predict the future position and trajectory of the robot arm. LSTM formula: input gate: forget gate: Memory update: Cell status: Output gate: Hidden state: , weights and biases for the corresponding gates; : sigmoid activation function; : hyperbolic tangent activation function; Loss function: Use mean square error loss function for regression task: wherein, is the true value, is the predicted value, and N is the number of samples.

4. The deep reinforcement learning based manipulator force feedback control method of claim 3, wherein: The simulation test process in step four includes: Data preparation: Collect a large amount of gesture and robot arm motion data, and perform data preprocessing, and divide it into training set, validation set and test set. Model training: Use optimizer for back propagation and gradient descent, update model parameters; Cross-validation during training, adjust hyperparameters; Model validation and testing: Use validation set to evaluate the prediction accuracy of the model, adjust the model structure and hyperparameters; Test the generalization ability of the model on the test set, ensure its robustness in actual application.

5. The deep reinforcement learning based manipulator force feedback control method of claim 4, wherein: The specific process of convolutional neural network for gesture recognition in the simulation test process includes: Simulation data: generate simulated gesture image data, divided into training set and test set; Network architecture: includes input layer, convolutional layer, pooling layer, fully connected layer and output layer; Training: use cross-entropy loss function and Adam optimizer for training; Evaluation: evaluate the accuracy of the model on the test set; The specific process of robot arm motion trajectory prediction includes: Simulation data: generate robot arm motion trajectory data, including position, velocity and acceleration; Network architecture: LSTM model, including input layer, LSTM layer and output layer; Training: use mean square error loss function and Adam optimizer for training; Evaluation: evaluate the prediction error of the model on the test set. 6.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is characterized in that: The processor executes the computer program to realize the steps of the robot arm force feedback control method based on deep reinforcement learning in any one of claims 1-5.

7. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by the processor to realize the steps of the robot arm force feedback control method based on deep reinforcement learning in any one of claims 1-5.

Images