A robotic arm control method and system based on deep learning and human-computer interaction

Abstract

The present invention discloses a robotic arm control method and system based on deep learning and human-computer interaction, comprising: collecting images from intelligent manufacturing scenarios, identifying the position and posture of each target object in the images, and rendering the images to generate a user operation interface; based on the user's control intention for the target object, and by gazing at the target object in the scene to induce brain waves, the brain waves are decoded to obtain the user's operation intention; receiving the user's operation intention through the robotic arm controller, obtaining the operation trajectory of the robotic arm based on the motion planning method, and using the dynamic model to drive the various joints of the robotic arm to perform grasping and placing tasks according to the operation trajectory. By integrating brain-computer interface technology and deep learning algorithms, the present invention provides a new robotic arm control solution.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a robotic arm control method and system based on deep learning and human-computer interaction. Background Art

[0002] IoT systems have been widely adopted in the field of intelligent manufacturing, significantly improving production efficiency and precision by integrating various sensors, networks, and devices. Within these applications, robotic arm control based on human-machine interaction has become an important research direction, particularly in scenarios requiring high precision and flexible operation. Currently, robotic arm control mostly relies on manual remote control, which can be achieved through keystroke control, voice control, or follow-up control. In this model, the human acts as the "driver," manipulating the robotic arm to complete tasks similar to how they operate a vehicle or aircraft. However, due to the complexity of robotic arm operations in three-dimensional space, it is difficult for a human driver to complete the preset tasks in "driver" mode.

[0003] Recent trends in brain-computer interface (BCI) research have shown that human intentions can be translated into control commands for robotic arms in IoT systems by analyzing neural signals. Based on BCI technology, by recognizing human intentions, users can achieve more natural control of robotic arms using an embodied control mode, just as they would control their own limbs. BCI technology is a new form of human-computer interaction that has recently developed. Leveraging the human brain's natural cognitive and responsive capabilities, BCIs offer a novel approach to control. BCIs collect and record EEG signals and convert them into actionable commands.

[0004] However, in existing BCI-based robotic arm control systems, users operate a robotic arm in a three-dimensional scene in two dimensions. Users do not truly interact with the three-dimensional scene. Their attention is always focused on the visual stimulation of the two-dimensional screen, limiting their ability to analyze and judge the environmental information of the three-dimensional scene. This leads to unnatural human-computer interaction in existing BCI-based brain-controlled robotic arm systems, making them difficult to apply in real-world scenarios. Furthermore, due to the complexity of operating the robotic arm in three dimensions, users often need to perform complex conversion operations when using control commands based on human intention to operate the robotic arm. This means that in order to control the robotic arm to reach the predetermined target, the user needs to generate a large number of control commands based on human intention, which also increases the user's workload. Therefore, how to apply BCI technology to robotic arm operation in three-dimensional scenes is a challenging problem. Summary of the Invention

[0005] In order to solve the problems of unnatural interaction and heavy user burden in the existing brain-computer interface-based robotic arm control system, the present invention proposes a robotic arm control method and system based on deep learning and human-computer interaction to solve the above problems; at the same time, the present invention has important application value in the intelligent manufacturing scenario of the Internet of Things, and can provide a new solution for robotic arm operation in the three-dimensional manufacturing scenario of the Internet of Things, thereby promoting the integrated development of brain-computer interface and Internet of Things technology.

[0006] To achieve the above objectives, the present invention provides a robotic arm control method based on deep learning and human-computer interaction, comprising:

[0007] Collect images from intelligent manufacturing scenarios, identify the position and posture of each target in the image, render the image, and generate an operation interface for the user;

[0008] Based on the user's control intention towards the target object, brain waves are induced by gazing at the target object in the scene, and the brain waves are decoded to obtain the user's operation intention;

[0009] The user's operation intention is received through the robotic arm controller, and the operation trajectory of the robotic arm is obtained based on the motion planning method. According to the operation trajectory, the dynamic model is used to drive the various joints of the robotic arm to perform grasping and placing tasks.

[0010] Preferably, generating the user operation interface includes:

[0011] A camera is used to capture images in the intelligent manufacturing scene from the user's first-person perspective. The position of each target in the image is identified through a deep learning algorithm. A sinusoidal wave coded transparency effect is superimposed on each target in the scene, and the image is rendered. The image in the intelligent manufacturing scene with the superimposed sinusoidal wave coded transparency effect is presented to the user in real time, generating an operation interface for the user.

[0012] Preferably, the deep learning algorithm is a neural network model implemented based on a transfer learning method, wherein the neural network model receives a picture in the intelligent manufacturing scene, extracts features through VGG16, and then performs posture estimation through a translation branch and a rotation branch to identify the position and posture of the target object;

[0013] Among them, the translation branch is used for position estimation, and outputs a three-dimensional vector to represent the position of the object in three-dimensional space; the rotation branch is used for posture estimation, and outputs a four-dimensional vector to represent the quaternion rotation of the object; the translation branch consists of three fully connected layers, which map the feature vector to 256 dimensions and 64 dimensions respectively, and finally output a three-dimensional position vector; the rotation branch consists of three fully connected layers, outputs a four-dimensional quaternion vector, and normalizes the quaternion through a custom normalization layer.

[0014] Preferably, the transparency of the sinusoidal wave encoding is:

[0015] alpha(t)=0.5·sin(2πft+Δφ)+0.5

[0016] Where alpha(t) is the transparency of the target at time t, f is the frequency of the sine wave, and Δφ is the phase difference.

[0017] Preferably, decoding the brain waves based on an incremental autonomous learning method comprises:

[0018] Step 1: Preprocessing and windowing the brain waves;

[0019] Preprocessing of EEG, including baseline removal and bandpass filtering;

[0020] Remove the baseline using a high-pass filter, specifically:

[0021] Y(t)=HighpassFilter(X(t),f cutoff )

[0022] Among them, f cutoff is the cutoff frequency of the high-pass filter, X(t) is the original brain wave, Y(t) is the signal after high-pass filter filtering, and HighpassFilter() is high-pass filtering;

[0023] The process of bandpass filtering the brain waves is as follows:

[0024] Z(t)=BandstopFilter(Y(t),50Hz)

[0025] Among them, the signal after bandpass filtering is Z(t), and BandstopFilter() is bandstop filtering;

[0026] The preprocessed EEG is sliced ​​into preset time windows, and each window contains a certain number of sample points N, specifically:

[0027] Z k =[Z(t k ),Z(t k +1),...,Z(t k +N-1)]

[0028] Z k is the windowed signal;

[0029] Step 2: Construct the initial brainwave template;

[0030] The stimulation frequency f is encoded according to the sine wave on the target object.i , construct the stimulation frequency f i The corresponding sine and cosine reference signals are:

[0031]

[0032] Where M is the number of harmonics, i is the ID of the target object, and k is the discretized time point. is the stimulation frequency f i corresponding sine and cosine reference signals;

[0033] Step 3: Calculate the correlation value;

[0034] The windowed signal Z is filtered using a zero-phase type I Chebyshev filter. k Decomposed into several sub-bands

[0035] The standard canonical correlation analysis algorithm is applied to each sub-band component to obtain the correlation value between each sub-band component and the predefined reference signal, specifically:

[0036]

[0037] Among them, ρ k is the correlation value corresponding to the kth template signal, is the correlation value of the Nth sub-band;

[0038] ρ k N sub-band components Perform weighted square sum fusion, that is:

[0039]

[0040] in, is the correlation output result after weighted square sum fusion, is the correlation value of the Nth sub-band, w(n) is the weighting function;

[0041] The weighting function w(n) is defined as:

[0042] w(n)=n -a +b,n∈[1,N]

[0043] Where a and b are both constants.

[0044] Preferably, obtaining the user's operation intention includes:

[0045] Collect the brainwave data generated by the user. When the number of data corresponding to the frequency of the sine wave encoding on each target exceeds M, a batch of user-specific templates is obtained. The user-specific template is used to generate a new correlation value, which is expressed as Each time a new user-specific template is collected, the previous user-specific template is discarded;

[0046] Finally, we get S weighted correlation values ​​corresponding to the stimulation frequencies of the sinusoidal waves encoded on the S targets. Take the largest one The corresponding target is the identified operation intention of the user;

[0047] User-specific templates are defined as:

[0048]

[0049] Among them, M is the number of data collected by a batch of user-specific templates; C is the number of EEG signal channels, For user-specific templates, This is the EEG data of a single trial and a single channel;

[0050] The optimized correlation values ​​are:

[0051]

[0052] Among them, α is the update weight parameter, To optimize the correlation value;

[0053] The user's operation intention is:

[0054]

[0055] in, is the preset threshold, f target Target ID for identification, is the optimized correlation value corresponding to the Sth target.

[0056] Preferably, obtaining the movement trajectory of the robotic arm based on a motion planning method includes:

[0057] The operation trajectory is generated using quintic polynomial interpolation, and the deviation between the operation trajectory and the desired path is corrected through proportional-differential control. When the user continues to input the operation intention, the robot arm will continue to execute the path of autonomous motion planning; otherwise, the robot arm stops moving;

[0058] The method for generating the running trajectory is as follows:

[0059] q(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5

[0060] Where, the polynomial coefficients a0, a1, ..., a5 are solved by setting boundary conditions, and q(t) is the running trajectory;

[0061] Proportional-derivative control is used to correct deviations from the desired path by:

[0062]

[0063] Where τ is the control input, K p and K d are the proportional and differential gain matrices, q desired and are the desired joint positions and velocities, q actual and are the actual joint positions and velocities, respectively.

[0064] Preferably, using the dynamic model to drive the joints of the robotic arm to perform grasping and placing tasks includes:

[0065] Obtaining an overall transformation matrix based on the transformation matrices of the joints of the robotic arm, and describing the forces and moments that produce the desired joint motions using the dynamic model, including inertia, Coriolis force, and gravity effects;

[0066] Among them, the transformation matrix T of each joint i for:

[0067]

[0068] Where θ i is the joint angle, d i is the connecting rod displacement, a i is the connecting rod length, α i is the connecting rod torsion angle;

[0069] The overall transformation matrix T from the base to the end effector is obtained by multiplying the transformation matrices of each joint:

[0070] T=T1T2T3T4T5T6;

[0071] The Lagrange method is used to describe the kinetic model, specifically:

[0072]

[0073] Where τ is the joint torque vector, M(θ) is the joint space inertia matrix, is the Coriolis force and centrifugal force matrix, G(θ) is the gravitational moment vector, is the joint acceleration vector, is the joint velocity vector;

[0074] The joint space inertia matrix M(θ) represents the resistance of the robot's mass and configuration to acceleration, specifically:

[0075]

[0076] Where m 11 (θ) is the inertial coupling effect between joint 1 and joint 1;

[0077] The Coriolis and centrifugal force matrix Reflects the force acting on the robot when it moves, specifically:

[0078]

[0079] Where c 11 is the coupling effect of Coriolis force and centrifugal force between joint 1 and joint 1;

[0080] The gravity torque vector G(θ) represents the gravity torque acting on the robot link, specifically:

[0081]

[0082] Where g1(θ) is the gravitational torque on joint 1.

[0083] On the other hand, to achieve the above-mentioned purpose, the present invention also provides a robotic arm control system based on deep learning and human-computer interaction, comprising:

[0084] Operation interface generation module, user operation intention decoding module and robotic arm motion control module;

[0085] The operation interface generation module is used to present the screen of the intelligent manufacturing scene with the transparency effect of the superimposed sine wave encoding to the user in real time to generate an operation interface;

[0086] The user operation intention decoding module is used to decode brain waves using an incremental self-learning algorithm to obtain the user's operation intention;

[0087] The robotic arm motion control module is used to control the robotic arm to perform grasping and placing tasks.

[0088] Compared with the prior art, the present invention has the following advantages and technical effects:

[0089] (1) Enhanced naturalness of interaction

[0090] By using a camera to capture the user's first-person perspective and a deep learning algorithm to identify the target's position and posture, the system can seamlessly integrate the user's field of view with the robotic arm's operating environment. This embodied perception enhances the user's immersion and makes robotic arm operation more natural and intuitive.

[0091] The transparency effect of the sine wave encoding is superimposed on the target, so that multiple targets can be effectively distinguished in the same scene; this not only reduces visual fatigue, but also improves the user's target selection and operation accuracy, and reduces the complexity of operation.

[0092] (2) Reduce user workload

[0093] Traditional BCI-based robotic arm control systems often require lengthy training and calibration. However, this invention employs an incremental self-learning algorithm that enables the system to adapt to the user's operating habits and intentions in real time, reducing the user's learning burden. Furthermore, the algorithm can continuously optimize the decoding accuracy of brain waves, improving the system's response speed and accuracy.

[0094] Targets are automatically identified and labeled through deep learning technology, and this information is presented to users in real time. Users can trigger control intentions simply by looking at specific targets. Compared with traditional robotic arm control methods, this method greatly reduces the user's active operation burden.

[0095] (3) Improved system robustness and adaptability

[0096] Through the transparency effect of sinusoidal wave encoding, the system can simultaneously handle the operational requirements of multiple targets in complex intelligent manufacturing scenarios. This multi-target operation capability makes the system more flexible and efficient in dealing with multiple tasks and scenarios, and improves the robustness of the system.

[0097] Using deep learning algorithms to identify and locate targets in the scene in real time enables the system to accurately interpret environmental information in various complex scenarios and make optimal robotic arm operation decisions, further improving the system's adaptability.

[0098] (4) Promote the integration of brain-computer interface and Internet of Things technology

[0099] In the IoT smart manufacturing scenario, this invention provides a new robotic arm control solution by integrating brain-computer interface technology and deep learning algorithms. This integration promotes the application of brain-computer technology in the IoT and helps to achieve more intelligent and efficient manufacturing processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0101] Figure 1 This is a flow chart of a robotic arm control method based on deep learning and human-computer interaction according to an embodiment of the present invention;

[0102] Figure 2 Schematic diagram of the structure of a neural network model according to an embodiment of the present invention;

[0103] Figure 3 This is a schematic diagram of the incremental self-learning algorithm flow in an embodiment of the present invention;

[0104] Figure 4 Schematic diagram of the structure of a robotic arm control device according to an embodiment of the present invention;

[0105] Figure 5 A diagram of the acquisition channels used by participants in an online experiment according to an embodiment of the present invention;

[0106] Figure 6 This is a diagram of an experimental scenario in which a participant participates in an online experiment according to an embodiment of the present invention;

[0107] Figure 7 This is a graph showing the results of all volunteers completing tasks using three different robotic arm control devices according to an embodiment of the present invention. DETAILED DESCRIPTION

[0108] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0109] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0110] Example 1

[0111] This paper proposes a robot arm control method based on deep learning and human-computer interaction, such as Figure 1 ,include:

[0112] Collect images from intelligent manufacturing scenarios, identify the position and posture of each target in the image, render the image, and generate an operation interface for the user;

[0113] Based on the user's control intention towards the target object, the user induces brain waves by looking at the target object in the scene, decodes the brain waves, and obtains the user's operation interface;

[0114] The robot controller receives the user's operation intention, obtains the operation trajectory of the robot based on the motion planning method, and uses the dynamic model to drive the various joints of the robot to perform grasping and placing tasks according to the operation trajectory.

[0115] This invention provides a novel robotic arm control solution for IoT smart manufacturing scenarios by integrating brain-computer interface technology and deep learning algorithms. This integration promotes the application of brain-computer interface technology in the IoT, contributing to more intelligent and efficient manufacturing processes.

[0116] Furthermore, a user operation interface is generated, including:

[0117] A camera is used to capture images in the intelligent manufacturing scene from the user's first-person perspective. The position of each target in the image is identified through a deep learning algorithm. A sine wave coded transparency effect is superimposed on each target in the scene, and the image is rendered. The image in the intelligent manufacturing scene with the superimposed sine wave coded transparency effect is presented to the user in real time, generating a user operation interface.

[0118] Specifically, a camera is used to capture images in the smart manufacturing scene from the user's first-person perspective; a deep learning algorithm is used to identify the position (including position and posture) of each target in the image; a sinusoidal wave-coded transparency effect is superimposed on each target in the scene; the image is rendered, and the image in the smart manufacturing scene with the superimposed sinusoidal wave-coded transparency effect is presented to the user in real time.

[0119] By using a camera to capture the user's first-person perspective and employing deep learning algorithms to identify the target's position, the user's field of view is seamlessly integrated with the robotic arm's operating environment. This embodied perception enhances the user's sense of immersion and makes robotic arm operation more natural and intuitive. A sine wave-encoded transparency effect superimposed on the targets effectively distinguishes multiple targets in the same scene. This not only reduces visual fatigue but also improves target selection and operation accuracy, reducing operational complexity.

[0120] Furthermore, the deep learning algorithm is a neural network model based on the transfer learning method (such as Figure 2), the neural network model receives the picture in the intelligent manufacturing scene, extracts features through VGG16, and then performs posture estimation through the translation branch and the rotation branch respectively to identify the position of the target object; wherein, the translation branch is used for position estimation, and outputs a three-dimensional vector to represent the position of the object in three-dimensional space; the rotation branch is used for posture estimation, and outputs a four-dimensional vector to represent the quaternion rotation of the object; the translation branch is composed of three layers of fully connected layers, which map the feature vector to 256 dimensions and 64 dimensions respectively, and finally outputs a 3-dimensional position vector; the rotation branch is composed of three layers of fully connected layers, which outputs a 4-dimensional quaternion vector and normalizes the quaternion through a custom normalization layer. The neural network model receives the picture in the intelligent manufacturing scene, extracts features through VGG16, and then performs posture estimation through the translation branch and the rotation branch respectively to return the position of the target object.

[0121] Specifically, transfer learning is to improve the performance of the original model. This transfer learning-based method effectively utilizes the learning results of the pre-trained model on a large-scale dataset, which can improve the accuracy and robustness of pose estimation.

[0122] Furthermore, the transparency of the sine wave encoding is:

[0123] alpha(t)=0.5·sin(2πft+Δφ)+0.5

[0124] Where alpha(t) is the transparency of the target at time t, f is the frequency of the sine wave, and Δφ is the phase difference.

[0125] Sine wave encoding is used to superimpose a sine wave-encoded transparency on the target. Sine wave encoding can make the induced EEG smoother and more stable, thereby improving the EEG signal-to-noise ratio. This is crucial for practical EEG detection, as EEG waves are often noisy. Compared to traditional square wave or pulse wave modulation, sine wave modulation provides a softer visual stimulus, reducing visual fatigue for the user and making experiments or applications more comfortable.

[0126] Using transparency coding instead of brightness coding can reduce the intensity of direct visual stimulation, thereby reducing irritation to the user's eyes and potential visual fatigue. Transparency modulation allows the visual effects of multiple target objects to be superimposed without significantly changing the brightness. Therefore, when the transparency of different objects in the field of view is encoded with different sine wave frequencies, the user's brain can still detect different frequency responses, which makes multi-target detection possible. In scenes with large changes in ambient light intensity, the effect of transparency coding is relatively stable and not easily interfered by ambient light, so the system has high robustness.

[0127] Furthermore, the brain waves are decoded based on an incremental autonomous learning method, such as Figure 3 ,include:

[0128] Step 1: EEG signal preprocessing and windowing;

[0129] In this embodiment, based on the characteristics of EEG baseline drift and power frequency interference, the EEG waves need to be preprocessed first, including baseline removal and 50Hz bandpass filtering. Since the baseline drift of the EEG wave will affect the accuracy of the signal, it is necessary to remove the baseline drift first. Baseline removal is achieved by using a high-pass filter. Let the original EEG wave be X(t) and the signal after high-pass filtering be Y(t). The baseline removal process can be expressed as:

[0130] Y(t)=HighpassFilter(X(t),f cutoff )

[0131] Among them, f cutoff It is the cutoff frequency of the high-pass filter, usually set between 0.5Hz and 1Hz. HighpassFilter() is a high-pass filter.

[0132] In order to remove the 50Hz power frequency interference, the brain waves need to be bandpass filtered at 50Hz. The signal after bandpass filtering is Z(t), and the filtering process can be expressed as:

[0133] Z(t)=BandstopFilter(Y(t),50Hz)

[0134] Among them, BandstopFilter() is a band-stop filter.

[0135] The preprocessed brain waves are sliced ​​into certain time windows, and each window contains a certain number of sample points N for subsequent calculations.

[0136] Let the windowed signal be Z k , it can be expressed as:

[0137] Z k =[Z(t k ),Z(t k +1),...,Z(t k +N-1)].

[0138] Step 2: Construct the initial brainwave template;

[0139] The stimulation frequency f is encoded according to the sine wave on the target object. i , construct its corresponding sine and cosine reference signals:

[0140]

[0141] Where M is the number of harmonics, which is set to 5; i is the ID of the target object.

[0142] Step 3: Calculate the correlation value;

[0143] Use a zero-phase type I Chebyshev filter to transform the windowed signal into Z k Decomposed into several sub-bands

[0144] The standard canonical correlation analysis algorithm is applied to each sub-band component separately to obtain the correlation between each sub-band component and the predefined reference signal (corresponding to all stimulus frequencies). ) between the following values:

[0145]

[0146] Among them, the correlation value corresponding to the kth template signal is represented by the vector ρ k It indicates that it contains the correlation values ​​corresponding to N sub-bands.

[0147] ρ k N sub-band components Do a weighted square sum fusion, that is:

[0148]

[0149] Among them, the weighting function w(n) is defined as:

[0150] w(n)=n -a +b,n∈[1,N]

[0151] Among them, a and b are constants, and their values ​​are determined when the classifier performance reaches the best case.

[0152] Furthermore, obtaining the user's operation intention includes:

[0153] Collect the brainwave data generated by the user. When the number of data corresponding to the frequency of the sine wave encoding on each target exceeds M, a batch of user-specific templates is obtained. The user-specific template is used to generate a new correlation value, which is expressed as Each time a new user-specific template is collected, the previous user-specific template is discarded;

[0154] Finally, we get S weighted correlation values ​​corresponding to the stimulation frequencies of the sinusoidal waves encoded on the S targets. Take the largest one The corresponding target is the identified user's operational intention;

[0155] User-specific templates are defined as:

[0156]

[0157] Among them, M is the number of data collected by a batch of user-specific templates; C is the number of EEG signal channels, For user-specific templates, This is the EEG data of a single trial and a single channel;

[0158] The optimized correlation values ​​are:

[0159]

[0160] Among them, α is the update weight parameter, To optimize the correlation value;

[0161] The user's operation intention is:

[0162]

[0163] in, is the preset threshold, f target Target ID for identification, is the optimized correlation value corresponding to the Sth target.

[0164] Traditional brain-computer interface-based robotic arm control systems often require long periods of training and calibration, but the present invention uses an incremental self-learning algorithm that enables the system to adapt to the user's operating habits and intentions in real time, reducing the user's learning burden. At the same time, the algorithm can continuously optimize the decoding accuracy of brain waves, improving the system's response speed and accuracy. Through deep learning technology, the target is automatically identified and labeled, and this information is presented to the user in real time. The user only needs to look at a specific target to trigger the control intention. Compared with traditional robotic arm control methods, this method greatly reduces the user's active operation burden.

[0165] Furthermore, the trajectory of the robotic arm is obtained based on the motion planning method, including:

[0166] The running trajectory is generated using quintic polynomial interpolation, and the deviation between the running trajectory and the desired path is corrected through proportional-differential control. Only when the user continues to input the operation intention will the robotic arm continue to execute the path of autonomous motion planning; otherwise, the robotic arm stops moving. The motion planning algorithm is used to generate a smooth and collision-free running trajectory for the robotic arm to follow.

[0167] A smooth trajectory is generated using a quintic polynomial interpolation, expressed as:

[0168] q(t)=a0+a1t+a2t 2 +a3t 3 +a4t4 +a5t 5

[0169] Where, the polynomial coefficients a0, a1, ..., a5 are solved by setting boundary conditions, and q(t) is the running trajectory;

[0170] To ensure accurate execution of trajectory planning, proportional-derivative control is used to correct deviations from the desired path as follows:

[0171]

[0172] Where τ is the control input, K p and K d are the proportional and differential gain matrices, q desired and are the desired joint positions and velocities, q actual and are the actual joint positions and velocities, respectively.

[0173] Furthermore, the dynamic model is used to drive the joints of the robotic arm to perform grasping and placing tasks, including:

[0174] The overall transformation matrix is ​​obtained based on the transformation matrices of each joint of the manipulator, and the forces and torques that produce the required joint motion are described by the dynamic model, including inertia, Coriolis force and gravity effects;

[0175] Among them, the transformation matrix T of each joint i for:

[0176]

[0177] Where θ i is the joint angle, d i is the connecting rod displacement, a i is the connecting rod length, α i is the connecting rod torsion angle;

[0178] The overall transformation matrix T from the base to the end effector is obtained by multiplying the transformation matrices of each joint:

[0179] T=T1T2T3T4T5T6;

[0180] The Lagrange method is used to describe the kinetic model, specifically:

[0181]

[0182] Where τ is the joint torque vector, M(θ) is the joint space inertia matrix, is the Coriolis force and centrifugal force matrix, G(θ) is the gravitational moment vector, is the joint acceleration vector, is the joint velocity vector;

[0183] The joint space inertia matrix M(θ) represents the resistance of the robot's mass and configuration to acceleration and is specifically:

[0184]

[0185] Where m 11 (θ) is the inertial coupling effect between joint 1 and joint 1; if m 12 , represents joint 1 and joint 2, and so on.

[0186] Coriolis and Centrifugal Force Matrix Reflects the force acting on the robot when it moves, specifically:

[0187]

[0188] Where c 11 is the coupling effect of Coriolis force and centrifugal force between joint 1 and joint 1; if m 12 , represents joint 1 and joint 2, and so on.

[0189] The gravity torque vector G(θ) represents the gravity torque acting on the robot link, specifically:

[0190]

[0191] Where g1(θ) is the gravitational torque on joint 1.

[0192] Example 2

[0193] This embodiment provides a robotic arm control system based on deep learning and human-computer interaction, such as Figure 4 , including an operation interface generation module, a user operation intention decoding module and a robotic arm motion control module;

[0194] The operation interface generation module is used to present the screen of the intelligent manufacturing scene with the transparency effect of the superimposed sine wave encoding to the user in real time to generate an operation interface;

[0195] The user operation intention decoding module is used to decode brain waves using an incremental self-learning algorithm to obtain the user's operation intention;

[0196] The robotic arm motion control module is used to control the robotic arm to perform grasping and placing tasks.

[0197] The control system is further divided into a local client and a remote server. The local client and remote server communicate through a data distribution service. Deep learning algorithms and incremental self-learning algorithms, which consume computing resources, are deployed on the remote server. Motion planning algorithms, which require high real-time performance, are deployed on the local client. Users operate on the local client. The proposed system supports multiple local clients communicating with the same remote server.

[0198] Example 3:

[0199] In order to verify the effectiveness and implementation effect of the robotic arm control method and system based on deep learning and human-computer interaction proposed in this embodiment, the following scientific experiments were conducted.

[0200] (1) Experimental volunteers

[0201] The study recruited 12 healthy adult volunteers. All were right-handed, had no history of neurological disease, and had normal or corrected-to-normal vision. All volunteers signed informed consent forms and were fully briefed on the experimental procedures and potential risks. The study was reviewed by the institutional ethics committee.

[0202] (2) Experimental setup

[0203] In order to be closer to the daily use environment, the data collection of this experiment was not carried out in a shielded room, no electromagnetic interference shielding measures were taken, and the walking sounds of people around were not eliminated. The experimental operator introduced the objectives and content of the experiment to the volunteers. The volunteers sat in a comfortable chair with a monitor in front of them to present the user operation interface. The EEG signals were collected through the OpenBCI (https: / / openbci.com / ) amplifier device. To ensure signal quality, the skin resistance of all electrodes was kept below 10kΩ. The reference electrode is located at the left ear and the ground electrode is located at the top of the forehead. In addition to the reference electrode and the ground electrode, EEG data from 8 channels in the occipital region were also collected. The acquisition channel diagram used in the experiment is shown below Figure 5 shown.

[0204] (3) Online experimental process

[0205] The purpose of the online experiment is to evaluate the performance of a robotic arm control method and system based on deep learning and human-computer interaction disclosed in this invention.

[0206] The online experiment includes three different robotic arm control devices. The three different robotic arm control devices are the robotic arm control system (device 1) disclosed in this embodiment, the robotic arm control device based on two-dimensional dynamic stimulation (device 2) and the robotic arm control device based on two-dimensional fixed stimulation (device 3). Volunteers need to use these three different robotic arm control devices to control the corresponding three different devices to complete the experimental task. The experimental task is set up as follows: In an unstructured environment, a target object grasping and placement task is designed. In the scene, there are a total of three workpieces. At the beginning of the task, the three workpieces will appear randomly at any position on the table in the scene. Volunteers need to control the robotic arm to place the three workpieces to the specified position. There is no restriction on the operation order of the three workpieces. Each volunteer needs to complete five grasping tasks. During the online experiment, the volunteers' EEG data, task completion status, task completion time and robotic arm status data are collected.

[0207] The experimental scenario of volunteers participating in the online experiment is as follows Figure 6 shown.

[0208] (4) Experimental results

[0209] Figure 7 The average task completion time, average output latency, and average intention recognition ratio of all volunteers using three different robotic arm control devices to complete the task are shown.

[0210] The results showed that the average task completion time for all volunteers using the three different robotic arm control devices was 89.04s, 96.65s, and 104.39s, respectively. The average task completion time of the robotic arm control system based on the present invention was significantly lower than that of the robotic arm control device based on two-dimensional dynamic stimulation and the robotic arm control device based on two-dimensional fixed stimulation. Compared with the robotic arm control device based on two-dimensional dynamic stimulation and the robotic arm control device based on two-dimensional fixed stimulation, the average task completion time of the robotic arm control system based on the present invention was reduced by 7.86% and 14.70%, respectively. The average output delay of all volunteers using the three different robotic arm control devices to complete the task was 2.22s, 2.01s, and 2.08s, respectively. Based on the robotic arm control system disclosed by the present invention, there was no significant difference in the average output delay between the volunteers using the robotic arm control device based on two-dimensional dynamic stimulation and the robotic arm control device based on two-dimensional fixed stimulation to complete the task. The average intention recognition ratio of all volunteers using the three different robotic arm control devices to complete the task was 46.59%, 56.14%, and 63.51%. The robot control system disclosed in this invention achieved a significantly lower average intent recognition ratio than both robot control devices based on two-dimensional dynamic stimulation and those based on two-dimensional fixed stimulation. Compared to these two devices, the average intent recognition ratio for the robot control system disclosed in this invention was reduced by 17.01% and 26.65% respectively.

[0211] In summary, the robotic arm control device disclosed in this invention shortens task time, reduces user workload (lowers the intention recognition ratio), and provides users with a better operating experience. These results demonstrate the potential advantages of the robotic arm control device disclosed in this invention in practical applications.

[0212] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A robotic arm control method based on deep learning and human-computer interaction, characterized in that: include: Collect images from intelligent manufacturing scenarios, identify the position and posture of each target in the image, render the image, and generate an operation interface for the user; Based on the user's control intention towards the target object, brain waves induced by gazing at the target object in the scene are decoded to obtain the user's operation intention; The robot controller receives the user's operation intention, obtains the movement trajectory of the robot based on the motion planning method, and drives each joint of the robot using the dynamic model according to the movement trajectory to perform the grasping and placing tasks; Generating the user's operation interface includes: Using a camera to capture images in the smart manufacturing scene from the user's first-person perspective, identifying the position of each target in the image through a deep learning algorithm, superimposing a sine wave-coded transparency effect on each target in the scene, and rendering the image. The image in the smart manufacturing scene with the superimposed sine wave-coded transparency effect is presented to the user in real time to generate an operation interface for the user; The deep learning algorithm is a neural network model implemented based on a transfer learning method. The neural network model receives the image in the intelligent manufacturing scene, extracts features through VGG16, and then performs posture estimation through translation and rotation branches to identify the position and posture of the target. Among them, the translation branch is used for position estimation and outputs a three-dimensional vector to represent the position of the object in three-dimensional space; the rotation branch is used for posture estimation and outputs a four-dimensional vector to represent the quaternion rotation of the object; the translation branch consists of three fully connected layers, which map the feature vector to 256 dimensions and 64 dimensions respectively, and finally output a three-dimensional position vector; the rotation branch consists of three fully connected layers, outputs a four-dimensional quaternion vector, and normalizes the quaternion through a custom normalization layer; Obtaining the user's operation intention includes: Collect the brainwave data generated by the user. When the number of data corresponding to the frequency of the sine wave encoding on each target exceeds the number of harmonics M, a batch of user-specific templates is obtained. The user-specific template is used to generate a new correlation value, which is expressed as Each time a new user-specific template is collected, the previous user-specific template is discarded; Finally, we get S weighted correlation values ​​corresponding to the stimulation frequencies of the sinusoidal waves encoded on the S targets. Take the largest one The corresponding target is the identified operation intention of the user; User-specific templates are defined as: Among them, M1 is the number of data collected by a batch of user-specific templates; C is the number of EEG signal channels, For user-specific templates, This is the EEG data of a single trial and a single channel; The optimized correlation values ​​are: Among them, α is the update weight parameter, To optimize the correlation value; The user's operation intention is: in, is the preset threshold, f target Target ID for identification, is the optimized correlation value corresponding to the Sth target.

2. The robotic arm control method based on deep learning and human-computer interaction according to claim 1, characterized in that: The transparency of the sine wave encoding is: alpha(t)=0.5·sin(2πft+Δφ)+0.5 Where alpha(t) is the transparency of the target at time t, f is the frequency of the sine wave, and Δφ is the phase difference.

3. The robotic arm control method based on deep learning and human-computer interaction according to claim 1, characterized in that: Decoding the brainwaves based on an incremental autonomous learning method, comprising: Step 1: Preprocessing and windowing the brain waves; Preprocessing of EEG, including baseline removal and bandpass filtering; Remove the baseline using a high-pass filter, specifically: Y(t)=HighpassFilter(X(t),f cutoff ) Among them, f cutoff is the cutoff frequency of the high-pass filter, X(t) is the original brain wave, Y(t) is the signal after high-pass filter filtering, and HighpassFilter() is high-pass filtering; The process of bandpass filtering the brain waves is as follows: Z(t)=BandstopFilter(Y(t),50Hz) Among them, the signal after bandpass filtering is Z(t), and BandstopFilter() is bandstop filtering; The preprocessed EEG is sliced ​​into preset time windows, and each window contains a certain number of sample points N, specifically: Z k =[Z(t k ),Z(t k +1),...,Z(t k +N-1)] Z k is the windowed signal; Step 2: Construct the initial brainwave template; The stimulation frequency f is encoded according to the sine wave on the target object. i , construct the stimulation frequency f i The corresponding sine and cosine reference signals are: Where M is the number of harmonics, i is the ID of the target object, and k is the discretized time point. is the stimulation frequency f i corresponding sine and cosine reference signals; Step 3: Calculate the correlation value; The windowed signal Z is filtered using a zero-phase type I Chebyshev filter. k Decomposed into several sub-bands The standard canonical correlation analysis algorithm is applied to each sub-band component to obtain the correlation value between each sub-band component and the predefined reference signal, specifically: Among them, ρ k is the correlation value corresponding to the kth template signal, is the correlation value of the Nth sub-band; ρ k N sub-band components Perform weighted square sum fusion, that is: in, is the correlation output result after weighted square sum fusion, is the correlation value of the Nth sub-band, w(n) is the weighting function; The weighting function w(n) is defined as: w(n)=n -a +b,n∈[1,N] Where a and b are both constants.

4. The robotic arm control method based on deep learning and human-computer interaction according to claim 1, characterized in that: Obtaining the movement trajectory of the robotic arm based on a motion planning method includes: The operation trajectory is generated using quintic polynomial interpolation, and the deviation between the operation trajectory and the desired path is corrected through proportional-differential control. When the user continues to input the operation intention, the robot arm will continue to execute the path of autonomous motion planning; otherwise, the robot arm stops moving; The method for generating the running trajectory is as follows: q(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 Where, the polynomial coefficients a0, a1, ..., a5 are solved by setting boundary conditions, and q(t) is the running trajectory; Proportional-derivative control is used to correct deviations from the desired path by: Where τ is the control input, K p and K d are the proportional and differential gain matrices, q desired and are the desired joint positions and velocities, q actual and are the actual joint positions and velocities, respectively.

5. The robotic arm control method based on deep learning and human-computer interaction according to claim 1, characterized in that: The dynamic model is used to drive the joints of the robotic arm to perform grasping and placing tasks, including: Obtaining an overall transformation matrix based on the transformation matrices of the joints of the robotic arm, and describing the forces and moments that produce the desired joint motions using the dynamic model, including inertia, Coriolis force, and gravity effects; Among them, the transformation matrix T of each joint i for: Where θ i is the joint angle, d i is the connecting rod displacement, a i is the connecting rod length, α i is the connecting rod torsion angle; The overall transformation matrix T from the base to the end effector is obtained by multiplying the transformation matrices of each joint: T=T1T2T3T4T5T6; The Lagrange method is used to describe the kinetic model, specifically: Where χ is the joint torque vector, M(θ) is the joint space inertia matrix, is the Coriolis force and centrifugal force matrix, G(θ) is the gravitational moment vector, is the joint acceleration vector, is the joint velocity vector; The joint space inertia matrix M(θ) represents the resistance of the robot's mass and configuration to acceleration, specifically: Where m 11 (θ) is the inertial coupling effect between joint 1 and joint 1; The Coriolis and centrifugal force matrix Reflects the force acting on the robot when it moves, specifically: Where c 11 is the coupling effect of Coriolis force and centrifugal force between joint 1 and joint 1; The gravity torque vector G(θ) represents the gravity torque acting on the robot link, specifically: Where g1(θ) is the gravitational torque on joint 1.

6. A robotic arm control system based on deep learning and human-computer interaction, used to implement the robotic arm control method based on deep learning and human-computer interaction according to any one of claims 1 to 5, characterized in that: It includes an operation interface generation module, a user operation intention decoding module and a robotic arm motion control module; The operation interface generation module is used to present the screen of the intelligent manufacturing scene with the transparency effect of the superimposed sine wave encoding to the user in real time to generate an operation interface; The user operation intention decoding module is used to decode brain waves using an incremental self-learning algorithm to obtain the user's operation intention; The robotic arm motion control module is used to control the robotic arm to perform grasping and placing tasks.

Images