Mechanical arm control method and system based on deep learning and man-machine interaction
Through deep learning and human-computer interaction technology, the target position in the three-dimensional scene is identified and the brain waves are decoded, which solves the problem of unnatural operation and work burden of the brain-computer interface in the three-dimensional scene, and achieves a more natural and efficient The operation of the robotic arm promotes the integration of brain-computer interfaces and Internet of Things technology.
Patent Information
- Application Number
- CN202411353138.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-26
AI Technical Summary
The existing brain-computer interface-based robotic arm control system is unnatural in three-dimensional scenarios, and users need to generate a large number of control instructions, resulting in a large workload and is difficult to apply in actual scenarios.
Using deep learning and human-computer interaction methods, the user's first-person perspective images are collected through the camera, the target position is recognized using deep learning algorithms, and the transparency effect rendering interface is rendered through the sine wave encoding, combined with the incremental self-learning algorithm to decode. Brain waves generate operation intentions and drive the robotic arm to perform tasks.
It improves the naturalness and accuracy of operations, reduces the work burden of users, enhances the robustness and adaptability of the system, promotes the integration of brain-computer interfaces and Internet of Things technology, and realizes a more intelligent manufacturing process.
Smart Images

Figure CN119077737A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a robot arm control method and system based on deep learning and human-computer interaction. Background Art
[0002] IoT systems have been widely used in the field of intelligent manufacturing, greatly improving production efficiency and precision by integrating various sensors, networks and devices. Among these applications, robotic arm control based on human-computer interaction has become an important research direction, especially in scenarios that require high precision and flexible operation. At present, the control of robotic arms mostly relies on manual remote control, which includes key control, voice control or follow-up control. In this mode, humans play the role of "drivers", manipulating robotic arms to complete tasks similar to the mode of operating vehicles or aircraft. However, due to the complexity of the operation of robotic arms in three-dimensional space, it is difficult for human drivers to complete preset tasks in "driver" mode.
[0003] In recent years, the latest trends in brain-computer interface research have shown that human intentions can be converted into control instructions for the robotic arm in the Internet of Things system by analyzing neural signals. Based on brain-computer interface technology, by identifying human intentions, users can achieve more natural robotic arm operations in the "embodied" control mode, just like controlling their own limbs. Brain-computer interface technology is a new type of human-computer interaction developed in recent years. Brain-computer interface uses the natural advantages of the human brain in cognition and reaction to provide a new way of control. Brain-computer interface collects and records EEG signals and converts them into actionable instructions.
[0004] However, in the existing BCI-based robotic arm control systems, users operate the robotic arm in a three-dimensional scene in two-dimensional space. The user does not really interact with the three-dimensional scene. The user's attention is always focused on the visual stimulation of the two-dimensional screen, which limits the user's analysis and judgment through the environmental information of the three-dimensional scene. This leads to the problem of unnatural human-computer interaction in the existing brain-controlled robotic arm system based on BCI, which is difficult to be applied in actual scenes. Not only that, due to the complexity of the operation of the robotic arm in three-dimensional space, users often need to perform complex conversion operations when using control instructions based on human intentions to operate the robotic arm to move. This means that in order to control the robotic arm to reach the predetermined target, the user needs to generate a large number of control instructions based on human intentions, which also increases the user's workload. Therefore, how to apply brain-computer interface technology to robotic arm operations in three-dimensional scenes is a challenging problem. Summary of the invention
[0005] In order to solve the problems of unnatural interaction and heavy user burden in the existing brain-computer interface-based robotic arm control system, the present invention proposes a robotic arm control method and system based on deep learning and human-computer interaction to solve the above problems; at the same time, the present invention has important application value in the intelligent manufacturing scenarios of the Internet of Things, and can provide new solutions for robotic arm operations in three-dimensional manufacturing scenarios of the Internet of Things, thereby promoting the integrated development of brain-computer interface and Internet of Things technology.
[0006] To achieve the above object, the present invention provides a robot arm control method based on deep learning and human-computer interaction, comprising:
[0007] Collect images in the intelligent manufacturing scene, identify the position and posture of each target in the image, render the image, and generate an operation interface for the user;
[0008] Based on the control intention of the user to the target object, brain waves are induced by looking at the target object in the scene, and the brain waves are decoded to obtain the user's operation intention;
[0009] The user's operation intention is received through the robot arm controller, and the operation trajectory of the robot arm is obtained based on the motion planning method. According to the operation trajectory, the dynamic model is used to drive the various joints of the robot arm to perform grasping and placing tasks.
[0010] Preferably, generating the user operation interface includes:
[0011] A camera is used to capture images in the smart manufacturing scene from the user's first-person perspective, and the position of each target object in the image is identified through a deep learning algorithm. A sine wave coded transparency effect is superimposed on each target object in the scene, and the image is rendered. The image in the smart manufacturing scene with the sine wave coded transparency effect superimposed is presented to the user in real time, generating an operation interface for the user.
[0012] Preferably, the deep learning algorithm is a neural network model implemented based on a transfer learning method, the neural network model receives the picture in the intelligent manufacturing scene, extracts features through VGG16, and then performs posture estimation through a translation branch and a rotation branch to identify the position and posture of the target object;
[0013] Among them, the translation branch is used for position estimation, and outputs a three-dimensional vector to represent the position of the object in three-dimensional space; the rotation branch is used for posture estimation, and outputs a four-dimensional vector to represent the quaternion rotation of the object; the translation branch consists of three layers of fully connected layers, which map the feature vector to 256 dimensions and 64 dimensions respectively, and finally output a 3D position vector; the rotation branch consists of three layers of fully connected layers, outputs a 4-dimensional quaternion vector, and normalizes the quaternion through a custom normalization layer.
[0014] Preferably, the transparency of the sine wave encoding is:
[0015] alpha(t)=0.5·sin(2πft+Δφ)+0.5
[0016] Among them, alpha(t) is the transparency of the target at time t, f is the frequency of the sine wave, and Δφ is the phase difference.
[0017] Preferably, decoding the brain waves based on an incremental autonomous learning method comprises:
[0018] Step 1: preprocessing and windowing the brain waves;
[0019] Preprocessing of EEG, including baseline removal and bandpass filtering;
[0020] The baseline is removed by high-pass filtering, specifically:
[0021] Y(t)=HighpassFilter(X(t),f cutoff )
[0022] Among them, f cutoff is the cutoff frequency of the high-pass filter, X(t) is the original brain wave, Y(t) is the signal after high-pass filter filtering, HighpassFilter() is high-pass filtering;
[0023] The process of bandpass filtering the brain waves is:
[0024] Z(t)=BandstopFilter(Y(t),50Hz)
[0025] Among them, the signal after bandpass filtering is Z(t), and BandstopFilter() is bandstop filtering;
[0026] The preprocessed EEG is sliced according to the preset time window, and each window contains a certain number of sample points N, specifically:
[0027] Z k =[Z(t k ),Z(t k +1),...,Z(t k +N-1)]
[0028] Z k is the windowed signal;
[0029] Step 2: Construct the initial brain wave template;
[0030] The stimulation frequency f is encoded according to the sine wave on the target object.i , construct the stimulation frequency f i The corresponding sine and cosine reference signals are:
[0031]
[0032] Where M is the number of harmonics, i is the ID of the target object, and k is the discretized time point. is the stimulation frequency f i corresponding sine and cosine reference signals;
[0033] Step 3: Calculate the correlation value;
[0034] The windowed signal Z is filtered using a zero-phase type I Chebyshev filter. k Decompose into several sub-bands
[0035] The standard canonical correlation analysis algorithm is applied to each sub-band component to obtain the correlation value between each sub-band component and the predefined reference signal, specifically:
[0036]
[0037] Among them, ρ k is the correlation value corresponding to the kth template signal, is the correlation value of the Nth sub-band;
[0038] ρ k N sub-band components Perform weighted square sum fusion, that is:
[0039]
[0040] in, is the correlation output result after weighted square sum fusion, is the correlation value of the Nth sub-band, w(n) is the weighting function;
[0041] The weighting function w(n) is defined as:
[0042] w(n)=n -a +b,n∈[1,N]
[0043] Where a and b are both constants.
[0044] Preferably, obtaining the user's operation intention includes:
[0045] Collect the brain wave data generated by the user. When the number of data corresponding to the frequency of the sine wave encoding on each target exceeds M, a batch of user-specific templates is obtained. The user-specific template is used to generate a new correlation value, which is expressed as Wherein, each time a new user-specific template is collected, the previous user-specific template is discarded;
[0046] Finally, we obtain S weighted correlation values corresponding to the stimulation frequencies encoded by the sine waves on the S targets. Take the largest one The corresponding target object is the identified operation intention of the user;
[0047] User-specific templates are defined as:
[0048]
[0049] Where M is the number of data collected by a batch of user-specific templates; C is the number of EEG signal channels, For user-specific templates, This is the EEG data of a single trial and a single channel;
[0050] The optimized correlation values are:
[0051]
[0052] Among them, α is the update weight parameter, To optimize the correlation value;
[0053] The user's operation intention is:
[0054]
[0055] in, is the preset threshold, f target The target ID is identified. is the optimized correlation value corresponding to the Sth target.
[0056] Preferably, obtaining the running trajectory of the robotic arm based on a motion planning method includes:
[0057] The running trajectory is generated by using quintic polynomial interpolation, and the deviation between the running trajectory and the expected path is corrected by proportional-differential control. When the user continuously outputs the operation intention, the robot arm will continue to execute the path planned by the autonomous motion, otherwise, the robot arm stops moving;
[0058] The method for generating the running trajectory is as follows:
[0059] q(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5
[0060] In the formula, the polynomial coefficients a0, a1, ..., a5 are solved by setting boundary conditions, and q(t) is the running trajectory;
[0061] Proportional-derivative control is used to correct deviations from the desired path by:
[0062]
[0063] Where τ is the control input, K p and K d are the proportional and differential gain matrices, q desired and are the desired joint positions and velocities, q actual and are the actual joint positions and velocities, respectively.
[0064] Preferably, using the dynamic model to drive each joint of the robotic arm to perform a grasping and placing task includes:
[0065] Obtaining an overall transformation matrix based on the transformation matrices of the joints of the robotic arm, and describing the forces and moments that produce the desired joint motions through the dynamic model, including inertia, Coriolis force, and gravity effects;
[0066] Among them, the transformation matrix T of each joint i for:
[0067]
[0068] In the formula, θ i is the joint angle, d i is the connecting rod displacement, a i is the connecting rod length, α i is the connecting rod torsion angle;
[0069] The overall transformation matrix T from the base to the end effector is obtained by multiplying the transformation matrices of each joint:
[0070] T = T1T2T3T4T5T6;
[0071] The kinetic model is described using the Lagrange method, specifically:
[0072]
[0073] Where τ is the joint torque vector, M(θ) is the joint space inertia matrix, is the Coriolis force and centrifugal force matrix, G(θ) is the gravitational moment vector, is the joint acceleration vector, is the joint velocity vector;
[0074] The joint space inertia matrix M(θ) represents the resistance of the robot's mass and configuration to acceleration, specifically:
[0075]
[0076] In the formula, m 11 (θ) is the inertial coupling effect between joint 1 and joint 1;
[0077] The Coriolis and Centrifugal Force Matrix Reflects the force acting on the robot when it moves, specifically:
[0078]
[0079] In the formula, c 11 is the coupling effect of Coriolis force and centrifugal force between joint 1 and joint 1;
[0080] The gravity torque vector G(θ) represents the gravity torque acting on the robot link, specifically:
[0081]
[0082] Where g1(θ) is the gravitational torque on joint 1.
[0083] On the other hand, to achieve the above purpose, the present invention also provides a robotic arm control system based on deep learning and human-computer interaction, comprising:
[0084] Operation interface generation module, user operation intention decoding module and robotic arm motion control module;
[0085] The operation interface generation module is used to present the picture in the intelligent manufacturing scene with the transparency effect of the superimposed sine wave encoding to the user in real time to generate an operation interface;
[0086] The user operation intention decoding module is used to decode brain waves using an incremental self-learning algorithm to obtain the user's operation intention;
[0087] The robot arm motion control module is used to control the robot arm to perform grasping and placing tasks.
[0088] Compared with the prior art, the present invention has the following advantages and technical effects:
[0089] (1) Enhanced naturalness of interaction
[0090] The present invention uses a camera to capture the user's first-person perspective and uses a deep learning algorithm to identify the position and posture of the target. The system can seamlessly combine the user's field of view with the robot arm operating environment. This embodied perception enhances the user's sense of immersion and makes the robot arm operation more natural and intuitive.
[0091] The transparency effect of the sine wave coding is superimposed on the target, so that multiple targets can be effectively distinguished in the same scene; this not only reduces visual fatigue, but also improves the user's target selection and operation accuracy, and reduces the complexity of operation.
[0092] (2) Reduce user workload
[0093] Traditional BCI-based robotic arm control systems often require long periods of training and calibration, while the present invention uses an incremental self-learning algorithm, which enables the system to adapt to the user's operating habits and intentions in real time, reducing the user's learning burden; at the same time, the algorithm can continuously optimize the decoding accuracy of brain waves and improve the system's response speed and accuracy;
[0094] Deep learning technology is used to automatically identify and label targets, and this information is presented to users in real time. Users can trigger control intentions simply by looking at specific targets. Compared with traditional robotic arm control methods, this method greatly reduces the user's active operation burden.
[0095] (3) Improved system robustness and adaptability
[0096] Through the transparency effect of sine wave encoding, the system can simultaneously handle the operational requirements of multiple targets in complex intelligent manufacturing scenarios. This multi-target operation capability makes the system more flexible and efficient in dealing with multiple tasks and scenarios, and improves the robustness of the system.
[0097] The use of deep learning algorithms to identify and locate targets in the scene in real time enables the system to accurately interpret environmental information in various complex scenarios and make optimal robotic arm operation decisions, further improving the adaptability of the system.
[0098] (4) Promote the integration of brain-computer interface and Internet of Things technology
[0099] In the IoT smart manufacturing scenario, the present invention provides a new robotic arm control solution by integrating brain-computer interface technology and deep learning algorithms. This integration promotes the application of brain-computer technology in the IoT and helps to achieve a more intelligent and efficient manufacturing process. BRIEF DESCRIPTION OF THE DRAWINGS
[0100] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0101] Figure 1 This is a flow chart of a method for controlling a robotic arm based on deep learning and human-computer interaction according to an embodiment of the present invention;
[0102] Figure 2 A schematic diagram of the structure of a neural network model according to an embodiment of the present invention;
[0103] Figure 3 A schematic diagram of the incremental self-learning algorithm flow in an embodiment of the present invention;
[0104] Figure 4 A schematic diagram of the structure of a robotic arm control device according to an embodiment of the present invention;
[0105] Figure 5 A collection channel diagram used by participants in an online experiment according to an embodiment of the present invention;
[0106] Figure 6 An experimental scene diagram of a participant participating in an online experiment according to an embodiment of the present invention;
[0107] Figure 7 This is a graph showing the results of all volunteers completing tasks using three different robotic arm control devices according to an embodiment of the present invention. DETAILED DESCRIPTION
[0108] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0109] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0110] Embodiment 1
[0111] The present invention proposes a robot arm control method based on deep learning and human-computer interaction, such as Figure 1 ,include:
[0112] Collect images in the intelligent manufacturing scene, identify the position and posture of each target in the image, render the image, and generate an operation interface for the user;
[0113] Based on the user's control intention towards the target object, brain waves are induced by looking at the target object in the scene, and the brain waves are decoded to obtain the user's operation interface;
[0114] The user's operation intention is received through the robot arm controller, and the operation trajectory of the robot arm is obtained based on the motion planning method. According to the operation trajectory, the dynamic model is used to drive the various joints of the robot arm to perform grasping and placing tasks.
[0115] In the IoT smart manufacturing scenario, this invention provides a new robotic arm control solution by integrating brain-computer interface technology and deep learning algorithms. This integration promotes the application of brain-computer technology in the IoT and helps to achieve a more intelligent and efficient manufacturing process.
[0116] Furthermore, a user operation interface is generated, including:
[0117] A camera is used to capture images in the smart manufacturing scene from the user's first-person perspective, and the position of each target in the picture is identified through a deep learning algorithm. A sine wave-coded transparency effect is superimposed on each target in the scene, and the picture is rendered. The picture in the smart manufacturing scene with the superimposed sine wave-coded transparency effect is presented to the user in real time, generating a user operation interface.
[0118] Specifically, a camera is used to capture images in the smart manufacturing scene from the user's first-person perspective; a deep learning algorithm is used to identify the position (including position and posture) of each target object in the picture; a sine wave-coded transparency effect is superimposed on each target object in the scene; the picture is rendered, and the picture in the smart manufacturing scene with the superimposed sine wave-coded transparency effect is presented to the user in real time.
[0119] By using a camera to capture the user's first-person perspective and using a deep learning algorithm to identify the position of the target, the user's field of view can be seamlessly combined with the robot's operating environment. This embodied perception enhances the user's sense of immersion and makes the robot's operation more natural and intuitive. The target is superimposed with a sine wave-encoded transparency effect, so that multiple targets can be effectively distinguished in the same scene. This not only reduces visual fatigue, but also improves the user's target selection and operation accuracy, and reduces the complexity of the operation.
[0120] Furthermore, the deep learning algorithm is a neural network model implemented based on the transfer learning method (such as Figure 2), the neural network model receives the picture in the intelligent manufacturing scene, extracts features through VGG16, and then estimates the posture through the translation branch and the rotation branch to identify the position of the target object; wherein, the translation branch is used for position estimation, and outputs a three-dimensional vector to represent the position of the object in three-dimensional space; the rotation branch is used for posture estimation, and outputs a four-dimensional vector to represent the quaternion rotation of the object; the translation branch consists of three layers of fully connected layers, which map the feature vector to 256 dimensions and 64 dimensions respectively, and finally output a 3D position vector; the rotation branch consists of three layers of fully connected layers, outputs a 4-dimensional quaternion vector, and normalizes the quaternion through a custom normalization layer. The neural network model receives the picture in the intelligent manufacturing scene, extracts features through VGG16, and then estimates the posture through the translation branch and the rotation branch to return the position of the target object.
[0121] Specifically, transfer learning is to improve the performance of the original model. This method based on transfer learning effectively utilizes the learning results of the pre-trained model on a large-scale dataset and can improve the accuracy and robustness of posture estimation.
[0122] Furthermore, the transparency of the sine wave encoding is:
[0123] alpha(t)=0.5·sin(2πft+Δφ)+0.5
[0124] Among them, alpha(t) is the transparency of the target at time t, f is the frequency of the sine wave, and Δφ is the phase difference.
[0125] Sine wave coding is used to superimpose the transparency of sine wave coding on the target. Sine wave coding can make the induced brain waves smoother and more stable, thereby improving the signal-to-noise ratio of brain waves. This is very important for brain wave detection in practical applications because brain waves have a lot of noise components. Compared with traditional square wave or pulse wave modulation, the visual stimulation of sine wave modulation is softer, which will reduce the visual fatigue of the user, making the experiment or application more comfortable.
[0126] Using transparency coding instead of brightness coding can reduce the intensity of direct visual stimulation, thereby reducing stimulation to the user's eyes and potential visual fatigue. Modulation of transparency allows the visual effects of multiple target objects to be superimposed without significantly changing the brightness. Therefore, when the transparency of different objects in the field of view is encoded with different sine wave frequencies, the user's brain can still detect different frequency responses, which makes multi-target detection possible. In scenes with large changes in ambient light intensity, the effect of transparency coding is relatively stable and is not easily disturbed by ambient light, so the system has high robustness.
[0127] Further, the brain waves are decoded based on an incremental autonomous learning method, such as Figure 3 ,include:
[0128] Step 1: EEG signal preprocessing and windowing;
[0129] In this embodiment, according to the characteristics of brain wave baseline drift and power frequency interference, the brain wave needs to be preprocessed first, including baseline removal and 50Hz bandpass filtering. Since the baseline drift of the brain wave will affect the accuracy of the signal, it is necessary to remove the baseline drift first. Baseline removal is achieved by a high-pass filter. Let the original brain wave be X(t) and the signal filtered by the high-pass filter be Y(t), then the baseline removal process can be expressed as:
[0130] Y(t)=HighpassFilter(X(t),f cutoff )
[0131] Among them, f cutoff It is the cutoff frequency of the high-pass filter, usually set between 0.5Hz and 1Hz. HighpassFilter() is a high-pass filter.
[0132] In order to remove the 50Hz power frequency interference, the brain waves need to be bandpass filtered at 50Hz. The signal after bandpass filtering is Z(t), and the filtering process can be expressed as:
[0133] Z(t)=BandstopFilter(Y(t),50Hz)
[0134] Among them, BandstopFilter() is a band-stop filter.
[0135] The preprocessed EEG is sliced into certain time windows, each of which contains a certain number of sample points N for subsequent calculations.
[0136] Let the windowed signal be Z k , it can be expressed as:
[0137] Z k =[Z(t k ),Z(t k +1),...,Z(t k +N-1)].
[0138] Step 2: Construct the initial brain wave template;
[0139] The stimulation frequency f is encoded according to the sine wave on the target object. i , construct its corresponding sine and cosine reference signals:
[0140]
[0141] Wherein, M is the number of harmonics, which is set to 5; i is the ID of the target object.
[0142] Step 3: Calculate the correlation value;
[0143] The windowed signal is Z using a zero-phase type I Chebyshev filter k Decompose into several sub-bands
[0144] The standard canonical correlation analysis algorithm is applied to each sub-band component separately to obtain the correlation between each sub-band component and the predefined reference signal (corresponding to all stimulus frequencies). ) between:
[0145]
[0146] Among them, the correlation value corresponding to the kth template signal is represented by the vector ρ k It indicates that it contains the correlation values corresponding to N sub-bands.
[0147] ρ k N sub-band components Do a weighted square sum fusion, that is:
[0148]
[0149] Among them, the weighting function w(n) is defined as:
[0150] w(n)=n -a +b,n∈[1,N]
[0151] Among them, a and b are constants, and their values are determined when the classifier performance reaches the best case.
[0152] Furthermore, obtaining the user's operation intention includes:
[0153] Collect the brain wave data generated by the user. When the number of data corresponding to the frequency of the sine wave encoding on each target exceeds M, a batch of user-specific templates is obtained. The user-specific template is used to generate a new correlation value, which is expressed as Wherein, each time a new user-specific template is collected, the previous user-specific template is discarded;
[0154] Finally, we obtain S weighted correlation values corresponding to the stimulation frequencies encoded by the sine waves on the S targets. Take the largest one The corresponding target is the identified user's operation intention;
[0155] User-specific templates are defined as:
[0156]
[0157] Where M is the number of data collected by a batch of user-specific templates; C is the number of EEG signal channels, For user-specific templates, This is the EEG data of a single trial and a single channel;
[0158] The optimized correlation values are:
[0159]
[0160] Among them, α is the update weight parameter, To optimize the correlation value;
[0161] The user's operation intention is:
[0162]
[0163] in, is the preset threshold, f target The target ID is identified. is the optimized correlation value corresponding to the Sth target.
[0164] Traditional brain-computer interface-based robotic arm control systems often require long-term training and calibration, while the present invention adopts an incremental self-learning algorithm, which enables the system to adapt to the user's operating habits and intentions in real time, reducing the user's learning burden. At the same time, the algorithm can continuously optimize the decoding accuracy of brain waves and improve the response speed and accuracy of the system. The target is automatically identified and labeled through deep learning technology, and this information is presented to the user in real time. The user only needs to look at a specific target to trigger the control intention. Compared with the traditional robotic arm control method, this method greatly reduces the user's active operation burden.
[0165] Furthermore, the operation trajectory of the robot arm is obtained based on the motion planning method, including:
[0166] The running trajectory is generated by using quintic polynomial interpolation, and the deviation between the running trajectory and the expected path is corrected by proportional-differential control. When the user continuously outputs the operation intention, the robot arm will continue to execute the path of autonomous motion planning, otherwise, the robot arm stops moving. The motion planning algorithm is used to generate a smooth and collision-free running trajectory for the robot arm to follow.
[0167] A smooth trajectory is generated using a quintic polynomial interpolation, expressed as:
[0168] q(t)=a0+a1t+a2t 2 +a3t 3 +a4t4 +a5t 5
[0169] In the formula, the polynomial coefficients a0, a1, ..., a5 are solved by setting boundary conditions, and q(t) is the running trajectory;
[0170] To ensure accurate execution of trajectory planning, proportional-derivative control is used to correct deviations from the desired path as follows:
[0171]
[0172] Where τ is the control input, K p and K d are the proportional and differential gain matrices, q desired and are the desired joint positions and velocities, q actual and are the actual joint positions and velocities, respectively.
[0173] Furthermore, the dynamic model is used to drive the joints of the robot arm to perform grasping and placing tasks, including:
[0174] The overall transformation matrix is obtained based on the transformation matrices of each joint of the robot arm, and the forces and torques that produce the required joint motions are described by a dynamic model, including inertia, Coriolis force, and gravity effects;
[0175] Among them, the transformation matrix T of each joint i for:
[0176]
[0177] In the formula, θ i is the joint angle, d i is the connecting rod displacement, a i is the connecting rod length, α i is the connecting rod torsion angle;
[0178] The overall transformation matrix T from the base to the end effector is obtained by multiplying the transformation matrices of each joint:
[0179] T = T1T2T3T4T5T6;
[0180] The kinetic model is described using the Lagrange method, specifically:
[0181]
[0182] Where τ is the joint torque vector, M(θ) is the joint space inertia matrix, is the Coriolis force and centrifugal force matrix, G(θ) is the gravitational moment vector, is the joint acceleration vector, is the joint velocity vector;
[0183] The joint space inertia matrix M(θ) represents the resistance of the robot's mass and configuration to acceleration, specifically:
[0184]
[0185] In the formula, m 11 (θ) is the inertial coupling effect between joint 1 and joint 1; if m 12 , represents joint 1 and joint 2, and so on.
[0186] Coriolis and Centrifugal Force Matrix Reflects the force acting on the robot when it moves, specifically:
[0187]
[0188] In the formula, c 11 is the coupling effect of Coriolis force and centrifugal force between joint 1 and joint 1; if m 12 , represents joint 1 and joint 2, and so on.
[0189] The gravity torque vector G(θ) represents the gravity torque acting on the robot link, specifically:
[0190]
[0191] Where g1(θ) is the gravitational torque on joint 1.
[0192] Embodiment 2
[0193] This embodiment provides a robotic arm control system based on deep learning and human-computer interaction, such as Figure 4 , including an operation interface generation module, a user operation intention decoding module and a robotic arm motion control module;
[0194] The operation interface generation module is used to present the picture in the intelligent manufacturing scene with the transparency effect of the superimposed sine wave encoding to the user in real time to generate an operation interface;
[0195] The user operation intention decoding module is used to decode brain waves using an incremental self-learning algorithm to obtain the user's operation intention;
[0196] The robot arm motion control module is used to control the robot arm to perform grabbing and placing tasks.
[0197] The control system is also divided into a local client and a remote server. The local client and the remote server communicate through a data distribution service. The deep learning algorithm and incremental self-learning algorithm that consume computing resources are deployed on the remote server. The motion planning algorithm with high real-time requirements is deployed on the local client. The user operates on the local client. The proposed system supports multiple local clients to communicate with the same remote server.
[0198] Embodiment 3
[0199] In order to verify the effectiveness and implementation effect of the robotic arm control method and system based on deep learning and human-computer interaction proposed in this embodiment, the following scientific experiments were carried out.
[0200] (1) Experimental volunteers
[0201] The experiment recruited 12 healthy adult volunteers, all of whom were right-handed, had no history of neurological disease, and had normal or corrected vision. The volunteers signed informed consent forms before the experiment and learned in detail about the experimental process and its potential risks. The experiment has been reviewed by the ethics committee of this institution.
[0202] (2) Experimental setup
[0203] In order to be closer to the daily use environment, the data collection of this experiment was not carried out in a shielded room, and no electromagnetic interference shielding measures were taken, and the walking sounds of people around were not eliminated. The experimental operator introduced the objectives and content of the experiment to the volunteers. The volunteers sat in a comfortable chair with a monitor in front of them to present the user interface. EEG signals were collected through the OpenBCI (https: / / openbci.com / ) amplifier device. To ensure signal quality, the skin resistance of all electrodes was kept below 10kΩ. The reference electrode is located at the left ear and the ground electrode is located at the top of the forehead. In addition to the reference electrode and the ground electrode, 8 channels of EEG data from the occipital region were also collected. The acquisition channel diagram used in the experiment is shown below Figure 5 shown.
[0204] (3) Online Experiment Process
[0205] The purpose of the online experiment is to evaluate the performance of a robotic arm control method and system based on deep learning and human-computer interaction disclosed in the present invention.
[0206] The online experiment includes three different robotic arm control devices. The three different robotic arm control devices are the robotic arm control system (device 1) disclosed in this embodiment, the robotic arm control device based on two-dimensional dynamic stimulation (device 2) and the robotic arm control device based on two-dimensional fixed stimulation (device 3). Volunteers need to use these three different robotic arm control devices to control the corresponding three different devices to complete the experimental task. The experimental task is set as follows: In an unstructured environment, a target object grasping and placement task is designed. In the scene, there are a total of three workpieces. At the beginning of the task, the three workpieces will appear randomly at any position on the table in the scene. Volunteers need to control the robotic arm to place the three workpieces at the specified position. There is no restriction on the operation order of the three workpieces. Each volunteer needs to complete five grasping tasks. During the online experiment, the volunteers' EEG data, task completion status, task completion time and robotic arm status data are collected.
[0207] The experimental scenario of volunteers participating in the online experiment is as follows Figure 6 shown.
[0208] (4) Experimental results
[0209] Figure 7 The average task completion time, average output latency, and average intention recognition ratio for all volunteers completing the task using three different robotic arm control devices are shown.
[0210] The results show that the average task completion time for all volunteers to complete tasks using three different robotic arm control devices was 89.04s, 96.65s and 104.39s respectively. The average task completion time of the robotic arm control system disclosed by the present invention is significantly lower than that of the robotic arm control device based on two-dimensional dynamic stimulation and the robotic arm control device based on two-dimensional fixed stimulation. Compared with the robotic arm control device based on two-dimensional dynamic stimulation and the robotic arm control device based on two-dimensional fixed stimulation, the average task completion time of the robotic arm control system disclosed by the present invention is reduced by 7.86% and 14.70%. The average output delays of all volunteers to complete tasks using three different robotic arm control devices were 2.22s, 2.01s and 2.08s respectively. Based on the robotic arm control system disclosed by the present invention, there was no significant difference between the average output delays of volunteers to complete tasks using robotic arm control devices based on two-dimensional dynamic stimulation and robotic arm control devices based on two-dimensional fixed stimulation. The average intention recognition ratios of all volunteers to complete tasks using three different robotic arm control devices were 46.59%, 56.14% and 63.51%. The average intention recognition ratio of the robot control system disclosed in the present invention is significantly lower than that of the robot control device based on two-dimensional dynamic stimulation and the robot control device based on two-dimensional fixed stimulation. Compared with the robot control device based on two-dimensional dynamic stimulation and the robot control device based on two-dimensional fixed stimulation, the average intention recognition ratio of the robot control system disclosed in the present invention is reduced by 17.01% and 26.65%.
[0211] In summary, the robot arm control device disclosed in the present invention shortens the task time, reduces the user's workload (reduces the intention recognition ratio), and brings a better operating experience to the user. These results demonstrate the potential advantages of the robot arm control device disclosed in the present invention in practical applications.
[0212] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A robotic arm control method based on deep learning and human-computer interaction, characterized in that: include: Collect images in the intelligent manufacturing scene, identify the position and posture of each target in the image, render the image, and generate an operation interface for the user; Based on the control intention of the user to the target object, brain waves are induced by looking at the target object in the scene, and the brain waves are decoded to obtain the user's operation intention; The user's operation intention is received through the robot arm controller, and the operation trajectory of the robot arm is obtained based on the motion planning method. According to the operation trajectory, the dynamic model is used to drive the various joints of the robot arm to perform grasping and placing tasks.
2. The robot arm control method based on deep learning and human-computer interaction according to claim 1, characterized in that: Generating the user's operation interface includes: A camera is used to capture images in the smart manufacturing scene from the user's first-person perspective, and the position of each target object in the image is identified through a deep learning algorithm. A sine wave coded transparency effect is superimposed on each target object in the scene, and the image is rendered. The image in the smart manufacturing scene with the sine wave coded transparency effect superimposed is presented to the user in real time, generating an operation interface for the user.
3. The robot arm control method based on deep learning and human-computer interaction according to claim 2, characterized in that: The deep learning algorithm is a neural network model implemented based on a transfer learning method. The neural network model receives the picture in the intelligent manufacturing scene, extracts features through VGG16, and then performs posture estimation through a translation branch and a rotation branch to identify the position and posture of the target object; Among them, the translation branch is used for position estimation, and outputs a three-dimensional vector to represent the position of the object in three-dimensional space; the rotation branch is used for posture estimation, and outputs a four-dimensional vector to represent the quaternion rotation of the object; the translation branch consists of three layers of fully connected layers, which map the feature vector to 256 dimensions and 64 dimensions respectively, and finally output a 3D position vector; the rotation branch consists of three layers of fully connected layers, outputs a 4-dimensional quaternion vector, and normalizes the quaternion through a custom normalization layer.
4. The robot arm control method based on deep learning and human-computer interaction according to claim 2, characterized in that: The transparency of the sine wave encoding is: alpha(t)=0.5·sin(2πft+Δφ)+0.5 Among them, alpha(t) is the transparency of the target at time t, f is the frequency of the sine wave, and Δφ is the phase difference.
5. The robot arm control method based on deep learning and human-computer interaction according to claim 1, characterized in that: Decoding the brain waves based on an incremental autonomous learning method includes: Step 1: preprocessing and windowing the brain waves; Preprocessing of EEG, including baseline removal and bandpass filtering; The baseline is removed by high-pass filtering, specifically: Y(t)=HighpassFilter(X(t),f cutoff ) Among them, f cutoff is the cutoff frequency of the high-pass filter, X(t) is the original brain wave, Y(t) is the signal after high-pass filter filtering, HighpassFilter() is high-pass filtering; The process of bandpass filtering the brain waves is: Z(t)=BandstopFilter(Y(t),50Hz) Among them, the signal after bandpass filtering is Z(t), and BandstopFilter() is bandstop filtering; The preprocessed EEG is sliced according to the preset time window, and each window contains a certain number of sample points N, specifically: Z k =[Z(t k ),Z(t k +1),…,Z(t k +N-1)] Z k is the windowed signal; Step 2: Construct the initial brain wave template; The stimulation frequency f is encoded according to the sine wave on the target object. i , construct the stimulation frequency f i The corresponding sine and cosine reference signals are: Where M is the number of harmonics, i is the ID of the target object, and k is the discretized time point. is the stimulation frequency f i corresponding sine and cosine reference signals; Step 3: Calculate the correlation value; The windowed signal Z is filtered using a zero-phase type I Chebyshev filter. k Decompose into several sub-bands The standard canonical correlation analysis algorithm is applied to each sub-band component to obtain the correlation value between each sub-band component and the predefined reference signal, specifically: Among them, ρ k is the correlation value corresponding to the kth template signal, is the correlation value of the Nth sub-band; ρ k N sub-band components Perform weighted square sum fusion, that is: in, is the correlation output result after weighted square sum fusion, is the correlation value of the Nth sub-band, w(n) is the weighting function; The weighting function w(n) is defined as: w(n)=n -a +b,n∈[1,N] Where a and b are both constants.
6. The robot arm control method based on deep learning and human-computer interaction according to claim 5, characterized in that: Obtaining the user's operation intention, including: Collect the brain wave data generated by the user. When the number of data corresponding to the frequency of the sine wave encoding on each target exceeds M, a batch of user-specific templates is obtained. The user-specific template is used to generate a new correlation value, which is expressed as Wherein, each time a new user-specific template is collected, the previous user-specific template is discarded; Finally, we obtain S weighted correlation values corresponding to the stimulation frequencies encoded by the sine waves on the S targets. Take the largest one The corresponding target object is the identified operation intention of the user; User-specific templates are defined as: Where M is the number of data collected by a batch of user-specific templates; C is the number of EEG signal channels, For user-specific templates, This is the EEG data of a single trial and a single channel; The optimized correlation values are: Among them, α is the update weight parameter, To optimize the correlation value; The user's operation intention is: in, is the preset threshold, f target The target ID is identified. is the optimized correlation value corresponding to the Sth target.
7. The robot arm control method based on deep learning and human-computer interaction according to claim 1, characterized in that: Obtaining the running trajectory of the robotic arm based on a motion planning method includes: The running trajectory is generated by using quintic polynomial interpolation, and the deviation between the running trajectory and the expected path is corrected by proportional-differential control. When the user continuously outputs the operation intention, the robot arm will continue to execute the path planned by the autonomous motion, otherwise, the robot arm stops moving; The method for generating the running trajectory is as follows: q(t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 In the formula, the polynomial coefficients a0, a1, ..., a5 are solved by setting boundary conditions, and q(t) is the running trajectory; Proportional-derivative control is used to correct deviations from the desired path by: Where τ is the control input, K p and K d are the proportional and differential gain matrices, q desired and are the desired joint positions and velocities, q actual and are the actual joint positions and velocities, respectively.
8. The robot arm control method based on deep learning and human-computer interaction according to claim 1, characterized in that: The dynamic model is used to drive the joints of the robot arm to perform grasping and placing tasks, including: Obtaining an overall transformation matrix based on the transformation matrices of the joints of the robotic arm, and describing the forces and moments that produce the desired joint motions through the dynamic model, including inertia, Coriolis force, and gravity effects; Among them, the transformation matrix T of each joint i for: In the formula, θ i is the joint angle, d i is the connecting rod displacement, a i is the connecting rod length, α i is the connecting rod torsion angle; The overall transformation matrix T from the base to the end effector is obtained by multiplying the transformation matrices of each joint: T = T1T2T3T4T5T6; The kinetic model is described using the Lagrange method, specifically: Where τ is the joint torque vector, M(θ) is the joint space inertia matrix, is the Coriolis force and centrifugal force matrix, G(θ) is the gravitational moment vector, is the joint acceleration vector, is the joint velocity vector; The joint space inertia matrix M(θ) represents the resistance of the robot's mass and configuration to acceleration, specifically: In the formula, m 11 (θ) is the inertial coupling effect between joint 1 and joint 1; The Coriolis and Centrifugal Force Matrix Reflects the force acting on the robot when it moves, specifically: In the formula, c 11 is the coupling effect of Coriolis force and centrifugal force between joint 1 and joint 1; The gravity torque vector G(θ) represents the gravity torque acting on the robot link, specifically: Where g1(θ) is the gravitational torque on joint 1.
9. A robot arm control system based on deep learning and human-computer interaction, used to implement the robot arm control method based on deep learning and human-computer interaction according to any one of claims 1 to 8, characterized in that: It includes an operation interface generation module, a user operation intention decoding module and a robotic arm motion control module; The operation interface generation module is used to present the picture in the intelligent manufacturing scene with the transparency effect of the superimposed sine wave encoding to the user in real time to generate an operation interface; The user operation intention decoding module is used to decode brain waves using an incremental self-learning algorithm to obtain the user's operation intention; The robot arm motion control module is used to control the robot arm to perform grasping and placing tasks.
Citation Information
Patent Citations
Identification method of six-degree-of-freedom serial robot tail-end load kinetic parameters
CN108227492A
Mechanical arm autonomous grabbing method based on deep learning
CN110211180A
Power optimum based mechanical arm moving track planning method and device
CN111152212A
Parameter separation-based mechanical arm kinetic parameter identification method and device
CN116038681A
Brain-controlled hand exoskeleton method and device
CN118161317A
