Robotic arm control method, system and storage medium based on multi-modal drive

Abstract

The present application relates to the field of data processing technology, and discloses a method, system and storage medium for controlling a robotic arm based on multimodal drive. The method comprises: collecting data through a sensor module for preprocessing to obtain a multimodal data set; extracting features from the multimodal data set and performing feature fusion to obtain a fused feature vector; estimating the state data of the fused feature vector through an unscented Kalman filter, and performing state prediction through a recursive neural network to obtain the current state of the robotic arm and the predicted state of the robotic arm; performing robotic arm operation trajectory planning and analysis on the current state of the robotic arm and the predicted state of the robotic arm to obtain the target execution trajectory; generating a control strategy for the target execution trajectory through an event-triggered distributed coordinated control algorithm to obtain a real-time joint control instruction sequence, and controlling the robotic arm to perform operations through the real-time joint control instruction sequence. The present application improves the efficiency and accuracy of robotic arm control based on multimodal drive.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a robot arm control method, system and storage medium based on multi-modal drive. Background Art

[0002] In recent years, with the rapid development of intelligent manufacturing, multimodal-driven robotic arm control technology has been widely used in automated production, medical surgery assistance, space exploration and other fields. Traditional robotic arm control methods mainly rely on single-mode sensor data, such as pure visual servo or simple force feedback control, which often shows limitations when facing complex and changing working environments. In order to improve the flexibility, precision and environmental adaptability of robotic arms, researchers have begun to explore control strategies based on multimodal perception, which comprehensively utilizes multiple sensor data such as vision, force touch, joint encoders and inertial measurement units. This approach aims to mimic the ability of human operators to perform fine operations using multi-sensory information, enabling robotic arms to cope with various complex tasks more intelligently.

[0003] However, the existing multimodal robotic arm control methods still have some technical challenges that need to be addressed. First, the real-time acquisition, synchronization and preprocessing of multimodal data are complex and require the processing of a large amount of heterogeneous data, which places high demands on the real-time performance and computational efficiency of the system. Secondly, there are differences in time scale, spatial resolution and information expression between different modal data. How to effectively fuse these heterogeneous data and extract useful control information remains a difficult problem. In addition, in the face of dynamically changing environments and diverse task requirements, the adaptability and robustness of existing methods need to be improved. Especially when dealing with unknown objects or unstructured environments, control systems often find it difficult to quickly adjust strategies, resulting in performance degradation or even failure. Summary of the invention

[0004] The present application provides a robotic arm control method, system and storage medium based on multi-modal drive, which are used to improve the efficiency and accuracy of robotic arm control based on multi-modal drive.

[0005] In a first aspect, the present application provides a robotic arm control method based on multimodal drive, and the robotic arm control method based on multimodal drive includes: collecting data through a sensor module on the robotic arm and performing preprocessing to obtain a multimodal data set, wherein the multimodal data set includes: visual data, force-tactile data, joint encoder data and inertial measurement unit data; performing feature extraction on the multimodal data set and performing feature fusion through an attention mechanism to obtain a fused feature vector; performing state data estimation on the fused feature vector through an unscented Kalman filter, and performing state prediction through a recursive neural network to obtain the current state of the robotic arm and the predicted state of the robotic arm; performing robotic arm operation trajectory planning analysis on the current state of the robotic arm and the predicted state of the robotic arm to obtain a target execution trajectory; performing control strategy generation on the target execution trajectory through an event-triggered distributed coordinated control algorithm to obtain a real-time joint control instruction sequence, and controlling the robotic arm to perform operations through the real-time joint control instruction sequence.

[0006] In combination with the first aspect, in a first implementation method of the first aspect of the present application, the data is collected and preprocessed through the sensor module on the robotic arm to obtain a multimodal data set, including: high-speed video recording of the working space of the robotic arm to obtain a real-time image stream and a real-time video stream, and motion blur elimination processing is performed on the real-time image stream and the real-time video stream to obtain clear image data; the working space of the robotic arm is scanned by a depth camera to obtain three-dimensional point cloud data; the force-tactile sensor array of the end effector of the robotic arm is sampled at high frequency to obtain pressure distribution data, friction and sliding information, and the encoders of each joint of the robotic arm are collected to obtain joint angles, joint angular velocities and joint angular accelerations; data is collected from the inertial measurement unit installed on the robotic arm to obtain the linear acceleration and angular velocity of the robotic arm, and environmental data is collected through the temperature sensor and humidity sensor installed on the robotic arm to obtain the environment environmental parameter data; denoising the clear image data, the three-dimensional point cloud data and the environmental parameter data to obtain initial visual data; denoising the pressure distribution data, the friction force and the sliding information to obtain initial force-tactile data; denoising the joint angle, the joint angular velocity and the joint angular acceleration to obtain initial joint encoder data, and denoising the robotic arm linear acceleration and the robotic arm angular velocity to obtain initial inertial measurement unit data; merging the initial visual data, the initial force-tactile data and the initial inertial measurement unit data into denoised multimodal data, and performing outlier detection and elimination on the denoised multimodal data through a median filter to obtain smoothed multimodal data; performing time alignment on the smoothed multimodal data to obtain the multimodal data set, wherein the multimodal data set includes: visual data, force-tactile data, joint encoder data and inertial measurement unit data.

[0007] In combination with the first aspect, in a second implementation of the first aspect of the present application, the feature extraction of the multimodal data set and the feature fusion through the attention mechanism to obtain a fused feature vector include: performing spatial pyramid pooling on the visual data to obtain a multi-scale visual feature map, and performing adaptive feature selection on the multi-scale visual feature map to obtain a key visual feature descriptor; performing wavelet packet transform on the force-tactile data to obtain a time-frequency domain feature representation, and performing principal component analysis on the time-frequency domain feature representation to obtain target pressure distribution data and texture features; operating the joint encoder data and the inertial measurement unit data The forward and inverse kinematic solutions are calculated to obtain the posture features of the robot arm, and the differential dynamics modeling is performed on the posture features of the robot arm to obtain the dynamic features of the robot arm; the key visual feature descriptors, the target pressure distribution data, the texture features, the posture features of the robot arm and the dynamic features of the robot arm are time-aligned to obtain an aligned feature sequence; the aligned feature sequence is nonlinearly interpolated to obtain feature data of a uniform sampling rate; the feature data of the uniform sampling rate are importance-weighted through a cross-modal attention network to obtain a weighted feature representation, and the weighted feature representation is cascaded and fused at multiple levels to obtain the fused feature vector.

[0008] In combination with the first aspect, in a third implementation method of the first aspect of the present application, the state data of the fused feature vector is estimated by an unscented Kalman filter, and the state is predicted by a recursive neural network to obtain the current state of the robot and the predicted state of the robot, including: performing joint space mapping on the fused feature vector to obtain initial joint state estimation data, and performing kinematic forward solution calculation on the initial joint state estimation data to obtain end effector posture estimation data; performing object position estimation analysis on the end effector posture estimation data and force-tactile data to obtain target object position estimation data, wherein the target object position estimation data includes object center of mass coordinates and object contour data; the initial joint state estimation The system state vector is obtained by combining the data of the robot arm, the end-effector posture estimation data and the target object position estimation data; the system state vector is sampled by untraceable transformation to obtain a sigma point set, and the sigma point set is subjected to nonlinear state propagation to obtain prior state estimation data; the prior state estimation data is updated by using the fused feature vector to obtain the current state of the robot arm; the dynamic parameters of the current state of the robot arm are identified to obtain an adaptive dynamic model; the current state of the robot arm and the adaptive dynamic model are time-series modeled by a gated recurrent unit network to obtain a state evolution sequence; the state evolution sequence is subjected to a robot arm simulation operation analysis to obtain a predicted state of the robot arm.

[0009] In combination with the first aspect, in a fourth implementation of the first aspect of the present application, the robot arm operation trajectory planning analysis is performed on the current state of the robot arm and the predicted state of the robot arm to obtain the target execution trajectory, including: task decomposition of the current state of the robot arm to obtain a basic action unit sequence, and symbolic representation of the basic action unit sequence to obtain a task execution plan; three-dimensional reconstruction of the visual data in the multimodal data set to obtain dynamic environment point cloud data; probabilistic occupancy analysis of the dynamic environment point cloud data to obtain a spatial obstacle distribution map; and symbolic representation of the spatial obstacle distribution map. Perform timing prediction to obtain dynamic obstacle avoidance constraint data, and perform kinematic forward solution on the predicted state of the robot arm and the task execution plan to obtain the predicted trajectory of the end effector; perform image plane projection on the predicted trajectory of the end effector to obtain an image feature predicted trajectory, and perform sampling optimization on the image feature predicted trajectory and the dynamic obstacle avoidance constraint data to obtain a candidate trajectory set; perform multi-objective evaluation on the candidate trajectory set and the task execution plan to obtain a Pareto optimal trajectory set; perform dynamic constraint analysis on the Pareto optimal trajectory set and the current state of the robot arm to obtain the target execution trajectory.

[0010] In combination with the first aspect, in the fifth implementation method of the first aspect of the present application, the distributed coordinated control algorithm triggered by events generates a control strategy for the target execution trajectory to obtain a real-time joint control instruction sequence, and controls the robotic arm to perform operations through the real-time joint control instruction sequence, including: discretizing the target execution trajectory to obtain a key path point sequence, and performing inverse kinematics calculation on the key path point sequence to obtain an initial joint angle sequence; performing dynamic compensation on the initial joint angle sequence to obtain a compensated joint angle sequence; constructing event trigger conditions for the compensated joint angle sequence and the fused feature vector to obtain a trigger threshold set, and distributively configuring the trigger threshold set to obtain each joint of the robotic arm. Independent trigger conditions; reinforcement learning is performed on the compensation joint angle sequence and the independent trigger conditions of each joint of the robotic arm to obtain an optimized control strategy; multi-sampling rate decomposition is performed on the optimized control strategy and the trigger threshold set to obtain a hierarchical control instruction set of vision, force feedback and joint control; fault mode analysis is performed on the hierarchical control instruction set and the independent trigger conditions of each joint of the robotic arm to obtain a fault-tolerant control strategy library; human-computer interaction interface mapping is performed on the fault-tolerant control strategy library and the optimized control strategy to obtain a visual control panel; real-time data stream processing is performed on the visual control panel and the hierarchical control instruction set to obtain the real-time joint control instruction sequence, and the robotic arm is controlled to perform operations through the real-time joint control instruction sequence.

[0011] In combination with the first aspect, in a sixth implementation of the first aspect of the present application, the event trigger condition is constructed for the compensated joint angle sequence and the fused feature vector to obtain a trigger threshold set, and the trigger threshold set is distributedly configured to obtain independent trigger conditions for each joint of the robotic arm, including: performing joint space analysis on the compensated joint angle sequence to obtain the joint motion range; performing torque threshold calculation on the force-tactile data in the multimodal data set to obtain the joint torque range, and performing constraint analysis on the joint motion range and the joint torque range to obtain preliminary trigger conditions; performing visual analysis on the visual data in the multimodal data set to obtain the joint motion range; and performing constraint analysis on the visual data in the multimodal data set to obtain the joint torque range. The target object position is estimated to obtain the end effector position error, and the end effector position error is mapped to the Jacobian matrix to obtain the joint angle error threshold; the preliminary trigger condition and the joint angle error threshold are weightedly fused to obtain a comprehensive trigger threshold; the comprehensive trigger threshold is dynamically adjusted to calculate the coefficient to obtain an adaptive trigger threshold set; the adaptive trigger threshold set is analyzed for noise sensitivity to obtain an anti-interference trigger threshold; the anti-interference trigger threshold is analyzed for joint coupling to obtain the trigger threshold set, and the trigger threshold set is mapped to the robot arm topology structure to obtain an independent trigger condition for each joint of the robot arm.

[0012] In a second aspect, the present application provides a robotic arm control system based on multi-modal drive, the robotic arm control system based on multi-modal drive comprising:

[0013] A processing module, used for collecting data through a sensor module on the robotic arm and performing preprocessing to obtain a multimodal data set, wherein the multimodal data set includes: visual data, force-tactile data, joint encoder data and inertial measurement unit data;

[0014] A fusion module, used to extract features from the multimodal dataset and fuse the features through an attention mechanism to obtain a fused feature vector;

[0015] A prediction module is used to estimate the state data of the fused feature vector through an unscented Kalman filter and to predict the state through a recursive neural network to obtain the current state of the robot arm and the predicted state of the robot arm;

[0016] An analysis module is used to perform robot arm operation trajectory planning analysis on the current state of the robot arm and the predicted state of the robot arm to obtain a target execution trajectory;

[0017] A generation module is used to generate a control strategy for the target execution trajectory through an event-triggered distributed coordinated control algorithm, obtain a real-time joint control instruction sequence, and control the robot arm to perform operations through the real-time joint control instruction sequence.

[0018] A third aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned multi-modal drive-based robotic arm control method.

[0019] In the technical solution provided in this application, by designing an efficient multimodal data acquisition and preprocessing process, real-time synchronization and noise reduction of vision, force-tactile, joint encoder and inertial measurement unit data are achieved, which greatly improves data quality and processing efficiency, and lays a solid foundation for subsequent feature extraction and fusion; innovative spatial pyramid pooling and adaptive feature selection technology are adopted, combined with wavelet packet transform and principal component analysis, to achieve effective feature extraction of multimodal data, significantly enhancing the system's perception of the environment and tasks; by introducing cross-modal attention networks and multi-level cascade fusion mechanisms, this method successfully solves the problem of heterogeneous data fusion, so that information of different modalities can be organically combined to give full play to their respective advantages; based on the state estimation and prediction module of the unscented Kalman filter and recursive neural network, the system's current state and future trend of the robotic arm are greatly improved. The perception accuracy of potential provides a reliable basis for intelligent decision-making and control; by comprehensively considering the current state, predicted state and environmental constraints of the robot arm, combined with the fast random sampling tree algorithm and multi-objective optimization technology, this method can generate the optimal execution trajectory that not only meets the task requirements but also effectively avoids obstacles, significantly improving the motion planning ability and environmental adaptability of the robot arm; a distributed coordinated control algorithm based on event triggering is proposed, and the coordinated control of multiple joints is realized by designing adaptive trigger thresholds and independent trigger conditions, which not only reduces unnecessary communication and computing overhead, but also ensures the real-time and accuracy of control; through the efficient fusion of multimodal data, intelligent trajectory planning and adaptive control strategy generation, the operation accuracy, flexibility and task adaptability of the robot arm in complex dynamic environments are comprehensively improved, and the efficiency and accuracy of the robot arm control based on multimodal drive are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1 A schematic diagram of an embodiment of a method for controlling a robotic arm based on multi-modal drive in an embodiment of the present application;

[0022] Figure 2 This is a schematic diagram of an embodiment of a robotic arm control system based on multi-modal drive in an embodiment of the present application. DETAILED DESCRIPTION

[0023] Embodiments of the present application provide a method, system and storage medium for controlling a robotic arm based on multimodal drive. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 In the embodiment of the present application, an embodiment of the robot arm control method based on multi-modal drive includes:

[0025] Step S101: Collect data through the sensor module on the robot arm and perform preprocessing to obtain a multimodal data set, wherein the multimodal data set includes: visual data, force-tactile data, joint encoder data and inertial measurement unit data;

[0026] It is understandable that the execution subject of the present application may be a multi-modal driven mechanical arm control system, or a terminal or a server, which is not specifically limited here. The present application embodiment is described by taking the server as the execution subject as an example.

[0027] Specifically, a high-speed camera is used to shoot the workspace of the robot in real time to obtain image streams and video streams. These raw visual data usually contain motion blur, so a deblurring algorithm is needed to process them. Specifically, a deblurring method based on a convolutional neural network can be used, such as DeblurGAN, which effectively removes motion blur and improves image quality by learning the mapping relationship from blur to clarity through a generative adversarial network. At the same time, a depth camera is used to obtain three-dimensional point cloud data of the workspace, which provides important information for subsequent environmental modeling and obstacle detection. In terms of force-tactile data acquisition, the force-tactile sensor array installed on the end effector of the robot records the pressure distribution, friction and sliding information when in contact with the object at a high sampling rate (usually 1kHz). These data are crucial for accurately controlling the grasping force and judging the stability of the object. At the same time, the high-precision encoders of each joint of the robot collect joint angle, angular velocity and angular acceleration information in real time, providing basic data for kinematic and dynamic calculations. In addition, the inertial measurement unit (IMU) installed at the key parts of the robot collects linear acceleration and angular velocity data, which can be fused with the encoder data to provide more accurate posture estimation.

[0028] In order to adapt to different working environments, the robot arm is also equipped with temperature sensors and humidity sensors to collect environmental parameter data. These data can be used to adjust control parameters to adapt to the impact of temperature and humidity changes on the performance of the robot arm. All these raw data need to be denoised to reduce the impact of sensor noise and environmental interference. For visual data, wavelet transform is used for denoising, which can effectively retain image edge information while removing high-frequency noise. Force-tactile data and IMU data can use Kalman filter for noise suppression, which can comprehensively consider system model and measurement noise to provide optimal estimation. In the data preprocessing stage, median filter is used for outlier detection and removal. This method is particularly effective for removing instantaneous spike noise, and can maintain the overall trend of the data while removing outliers that are significantly deviated. Finally, considering that the sampling rates of different sensors may be different, time alignment processing is required. The dynamic time warping (DTW) algorithm plays an important role here. It can adaptively align time series data of different lengths and sampling rates to ensure the consistency of multimodal data in the time dimension.

[0029] Through the above steps, the final multimodal data set includes preprocessed visual data, force-tactile data, joint encoder data, and inertial measurement unit data. These data lay the foundation for subsequent feature extraction and fusion, ensuring that the control system can obtain high-quality, synchronized multimodal perception information.

[0030] For example, in a scenario where a robotic arm performs a precision assembly task, a high-speed camera collects images of the working area at a rate of 120 frames per second, and there is slight motion blur in the original image. After processing with DeblurGAN, the image clarity is improved by about 40% and the edge sharpness is increased by 25%. The point cloud data simultaneously collected by the depth camera has a resolution of 640x480, and each frame contains an average of about 300,000 three-dimensional points. The force-tactile sensor records the force data of the end effector contacting the part at a sampling rate of 1kHz, and there is about ±0.05N random noise in the original data. After Kalman filtering, the noise level is reduced to ±0.01N, significantly improving the accuracy of force feedback. The joint encoder collects data at a frequency of 500Hz, and the angular resolution reaches 0.01 degrees. The IMU data sampling rate is 200Hz, and the original data has about ±0.1m / s² acceleration noise and ±0.05° / s angular velocity noise. After filtering, these noises are reduced to ±0.02m / s² and ±0.01° / s, respectively. Environmental sensors recorded a temperature of 23 ± 0.5 ° C and a relative humidity of 45 ± 2% in the working area. All these data were time-aligned to form a unified multimodal dataset with a sampling rate of 100 Hz, providing rich and accurate input information for subsequent feature extraction and control decisions.

[0031] Step S102: extract features from the multimodal data set and fuse the features through an attention mechanism to obtain a fused feature vector;

[0032] Specifically, the visual data is processed by spatial pyramid pooling, which divides the image into grids of different scales and performs pooling operations within each grid to obtain multi-scale visual feature maps. Specifically, spatial pyramid pooling usually includes three levels: 1x1, 2x2, and 4x4 grid divisions, and maximum pooling operations are performed within each grid. This method can capture the global and local features of the image and improve the scale invariance of the features. Adaptive feature selection is performed on the multi-scale visual feature maps. This step uses a feature selection algorithm based on the channel attention mechanism, such as the Squeeze-and-Excitation (SE) module. The SE module first performs global average pooling on the feature map to obtain a channel descriptor, then learns the correlation between channels through a two-layer fully connected network, and finally reweights the original feature map with the learned weights. This process can adaptively emphasize important visual features and suppress unimportant features, thereby obtaining key visual feature descriptors.

[0033] For force-tactile data, wavelet packet transform is used for time-frequency analysis. Wavelet packet transform is an extension of wavelet transform. It performs multi-layer decomposition on the signal, not only decomposing the low-frequency part but also the high-frequency part, so it can provide finer time-frequency resolution. In this method, Daubechies wavelet basis is used for 5-layer decomposition to obtain time-frequency features of 32 frequency bands. Then, principal component analysis (PCA) is performed on these time-frequency features, and the principal components that explain 95% of the variance are retained to obtain compressed pressure distribution and texture feature representation. The joint encoder data and inertial measurement unit data are first calculated by kinematic forward and inverse solutions to obtain the posture characteristics of the manipulator. Forward kinematics uses the DH parameter method to calculate the position and posture of the end effector, while inverse kinematics uses an analytical method combined with a numerical iteration method to solve the joint angle. Then, the dynamic model of the manipulator is established using the Lagrange equation, and the dynamic characteristics of the manipulator are obtained by differential dynamics modeling, including joint torque, inertia matrix, and Coriolis force.

[0034] In order to align the feature sequences of different modalities, the dynamic time warping (DTW) algorithm is used. DTW finds the optimal alignment path between two time series through dynamic programming and can handle sequences of different lengths and sampling rates. In this method, the feature sequences of other modalities are aligned with the visual feature sequence as a benchmark. The aligned feature sequence is resampled by cubic spline interpolation to obtain feature data with a uniform sampling rate. The feature data with a uniform sampling rate is weighted by importance through a cross-modal attention network. The network adopts the Transformer architecture, which includes a multi-head self-attention mechanism and a cross-modal attention mechanism. The self-attention mechanism is used to capture the temporal dependencies within a single modality, while the cross-modal attention mechanism is used to learn the correlation between different modalities. Specifically, the features of each modality first pass through the self-attention layer, then perform cross-modal attention calculations with the features of other modalities, and finally obtain the weighted feature representation through a feedforward neural network.

[0035] The weighted feature representation is fused through multi-level cascade to form the final fused feature vector. Cascade fusion includes three levels: early fusion, mid-term fusion, and late fusion. Early fusion directly splices low-level features of different modalities; mid-term fusion uses convolutional neural networks to fuse intermediate features; and late fusion uses fully connected layers to fuse high-level features. This multi-level fusion strategy can make full use of feature information at different levels and improve the richness and robustness of feature representation.

[0036] For example, in a precision assembly task, a robot arm needs to insert a microelectronic component into a circuit board. First, the visual data is spatially pyramid pooled to obtain feature maps of three scales: 1x1, 2x2, and 4x4, each containing 256 channels. Through the adaptive feature selection of the SE module, the importance weights of the 256 channels are adjusted, with the weight of the most important channel reaching 0.95 and the weight of the least important channel reduced to 0.05. The force-tactile data is transformed by wavelet packet transform to obtain the time-frequency features of 32 frequency bands. After principal component analysis, the first 10 principal components are retained, explaining 97.8% of the variance. The position accuracy of the end effector obtained by kinematic calculation of the joint encoder and IMU data is ±0.01mm, and the attitude accuracy is ±0.1 degrees. The joint torque estimation error obtained by dynamic modeling is less than 2%. The DTW algorithm aligns the feature sequences of different modes, and the maximum time deviation is reduced from the original 200ms to 5ms. In the cross-modal attention network, the self-attention mechanism captures the importance score of key frames in the visual feature sequence to 0.8, while the ordinary frame is only 0.2. Cross-modal attention shows that at the critical moment of the insertion operation, the weight of the force-tactile feature rises to 0.6, while the weight of the visual feature drops to 0.3. After multi-level cascade fusion, a 1024-dimensional fused feature vector is finally obtained, which contains a comprehensive representation of visual, force-tactile, kinematic and dynamic information. This fused feature vector provides rich and accurate input information for subsequent state estimation and control decisions, enabling the robot arm to complete the task of inserting micro-components with an accuracy of 0.1mm and a force control accuracy of 0.5N.

[0037] Step S103, estimating the state data of the fused feature vector by using an unscented Kalman filter, and predicting the state by using a recursive neural network to obtain the current state of the robotic arm and the predicted state of the robotic arm;

[0038] Specifically, the fused feature vector is mapped to the joint space, and the high-dimensional feature vector is converted into an initial joint state estimate. This process is completed by a fully connected neural network, which maps the 1024-dimensional fused feature vector to the joint space of the robot arm to obtain an initial state vector containing joint angles, angular velocity, and angular acceleration. The kinematic model of the robot arm is used to perform forward kinematic calculations on the initial joint state estimate to obtain the pose estimate of the end effector. This step uses the DH parameter method to convert the joint angles into the position and pose of the end effector in Cartesian space through continuous matrix transformation. At the same time, the target object position is estimated in combination with force-tactile data. This process uses the force feedback information when the end effector contacts the object, combined with the target object contour information in the visual data, and estimates the center of mass coordinates and pose of the object through the least squares method.

[0039] The initial joint state estimate, end effector pose estimate, and target object position estimate are combined to form a complete system state vector. This state vector contains the internal state (joint state) and external state (position relationship between the end effector and the target object) of the robot, providing a comprehensive information basis for subsequent state estimation and prediction. The unscented Kalman filter (UKF) is a state estimation method suitable for nonlinear systems. It approximates the state distribution through a set of carefully selected sampling points (called sigma points). In this method, the implementation steps of UKF are as follows: First, the system state vector is sampled through an unscented transformation to generate 2n+1 sigma points, where n is the dimension of the state vector. Each sigma point contains a possible value of the state variable. Then, these sigma points are propagated through the system model for nonlinear state to obtain a priori state estimates. This process takes into account the dynamic model of the robot, including factors such as joint torque, inertia matrix, and Coriolis force.

[0040] The prior state estimate is updated using the fused feature vector as the measurement input. The UKF calculates the predicted measurement values ​​corresponding to each sigma point, and then compares these predicted measurements with the actual fused feature vector to calculate the Kalman gain. Finally, the Kalman gain is used to correct the prior estimate to obtain the current state estimate of the manipulator. This process can effectively fuse multimodal sensor information and improve the accuracy and robustness of state estimation. For state prediction, this method uses a recurrent neural network (RNN), especially a long short-term memory network (LSTM). LSTM is a variant of RNN that can effectively handle long-term dependencies. It controls the flow of information and the update of memory through a gating mechanism. Specifically, the LSTM network receives the current state of the manipulator and an adaptive dynamics model as input, and outputs a sequence of state predictions for a period of time in the future.

[0041] The training process of the LSTM network is as follows: First, a large amount of historical state sequence data is collected, including the motion data of the robot arm under different tasks and environmental conditions. Then, this data is divided into input sequence and target sequence in chronological order. The input sequence contains the state information of t time steps, and the target sequence contains the state information of the future k time steps. The LSTM network is trained using the back propagation through time (BPTT) algorithm to minimize the mean square error between the predicted value and the actual value. After training, the LSTM network can predict the state changes of the robot arm in the next few time steps based on the current state and dynamic model.

[0042] For example, in a scenario where a 6-DOF robot performs precision assembly tasks, the fused feature vectors are mapped into the joint space to obtain the initial joint state estimate, including the angles, angular velocities, and angular accelerations of the 6 joints, for a total of 18 variables. The kinematics forward solution calculates the position accuracy of the end effector to be ±0.05 mm and the attitude accuracy to be ±0.1 degrees. The accuracy of the target object position estimate reaches ±0.1 mm. The unscented Kalman filter uses 13 sigma points (2×6+1, with a state vector dimension of 6) for state estimation. The diagonal elements of the initial state covariance matrix are set to [0.01, 0.01, 0.01, 0.001, 0.001, 0.001], indicating the initial uncertainty of the position and attitude. The diagonal elements of the process noise covariance matrix Q are set to [1e-6, 1e-6, 1e-6, 1e-4, 1e-4, 1e-4], and the diagonal elements of the measurement noise covariance matrix R are set to [1e-4, 1e-4, 1e-4, 1e-3, 1e-3, 1e-3]. After UKF processing, the accuracy of state estimation is improved to ±0.02 mm for position and ±0.05 degrees for attitude.

[0043] The LSTM network has a 3-layer structure, with 128 hidden units in each layer. The input sequence length is 20 time steps (equivalent to 0.2 seconds of data, assuming a sampling rate of 100Hz), and the output predicts the state of the next 10 time steps. The network is trained for 50 rounds on a dataset containing 1 million samples, using the Adam optimizer with a learning rate of 0.001. After training, the LSTM network is able to predict the state of the robot arm within 0.1 seconds in the future with an R² score of 0.98.

[0044] In actual operation, when the robot needs to insert a precision part into a hole, UKF first estimates the current state, and LSTM then predicts the state change within the next 0.1 second. For example, during the insertion process, UKF estimates that the current end effector is 2.5mm away from the target hole, and the posture deviation is 0.8 degrees. LSTM predicts that in the next 0.1 seconds, the end effector will move 2.3mm and the posture deviation will be reduced to 0.3 degrees. These precise state estimation and prediction information provide an important basis for subsequent trajectory planning and control strategy generation, enabling the robot to complete the insertion task with sub-millimeter accuracy.

[0045] Step S104, performing robot arm operation trajectory planning analysis on the current state of the robot arm and the predicted state of the robot arm to obtain a target execution trajectory;

[0046] Specifically, the current state of the robot arm is task-decomposed, and the complex operation task is decomposed into a series of basic action units. This process uses a hierarchical task planning method to gradually refine high-level task goals into executable basic actions. For example, a task of "picking up an object and placing it" can be decomposed into basic action units such as "moving near the object", "adjusting the posture", "grasping the object", "moving to the target position" and "releasing the object".

[0047] Next, these basic action units are symbolically represented to obtain a high-level task execution plan. The symbolic representation uses predicate logic to represent each action unit as an operator with preconditions and postconditions. For example, the "grasp an object" operation can be represented as: GRASP(Object):

[0048] Precondition:At(HandPosition,ObjectPosition);HandEmpty;

[0049] Effect: Holding(Object); ¬HandEmpty

[0050] In this representation:

[0051] GRASP(Object) is the name of the operation, where Object is the object to be grasped.

[0052] Precondition lists the preconditions required to perform this operation:

[0053] At(HandPosition, ObjectPosition) means that the position of the hand is the same as the object position.

[0054] HandEmpty indicates that the robot hand is currently empty (not grasping other objects).

[0055] Effect describes the result of the operation:

[0056] Holding(Object) indicates that the robot is currently holding the object.

[0057] ¬HandEmpty indicates that the hand is no longer empty (the ¬ sign indicates negation).

[0058] This symbolic representation makes it easier for subsequent planning algorithms to understand and process task requirements.

[0059] Subsequently, the visual data in the multimodal dataset is used for 3D reconstruction to obtain dynamic environment point cloud data. This step uses structured light or time-of-flight (ToF) depth cameras combined with SLAM (simultaneous localization and mapping) technology to generate a high-precision 3D environment model. Point cloud data usually contains hundreds of thousands of 3D points, each of which contains spatial coordinates (x, y, z) and possible color information (R, G, B). Probabilistic occupancy analysis is performed on the obtained dynamic environment point cloud data to generate a spatial obstacle distribution map. This process uses the Octree data structure to efficiently represent and update occupancy probabilities in 3D space. Each voxel (3D pixel) has an occupancy probability value ranging from 0 (free) to 1 (occupied). These probability values ​​are continuously updated based on new sensor data through the Bayesian update rule.

[0060] In order to deal with moving obstacles in a dynamic environment, the spatial obstacle distribution map is predicted in time series to obtain dynamic obstacle avoidance constraint data. Here, the Kalman filter is used to predict the trajectory of the moving object, and the prediction time range is usually 0.5-1 second in the future. The prediction results are used to update the spatial occupancy probability at future time points. The predicted state of the robot and the task execution plan are kinematically solved to obtain the predicted trajectory of the end effector. This step uses the DH parameters and joint angle prediction sequence of the robot to calculate the position and posture sequence of the end effector in Cartesian space.

[0061] The predicted trajectory of the end effector is projected onto the image plane to obtain the predicted trajectory of the image features. This process uses the camera's intrinsic and extrinsic parameter matrices to convert the trajectory points in 3D space into 2D image coordinates. This representation facilitates direct comparison and optimization with visual feedback.

[0062] Next, the rapidly expanding random tree (RRT) algorithm is used to sample and optimize the image feature prediction trajectory and dynamic obstacle avoidance constraint data to obtain a set of candidate trajectories. The RRT algorithm randomly samples in the configuration space to quickly explore feasible paths while ensuring asymptotic optimality. In this method, the sampling strategy of RRT is modified to favor the area near the image feature prediction trajectory to improve planning efficiency.

[0063] A multi-objective evaluation is performed on the obtained candidate trajectory set and task execution plan to obtain the Pareto optimal trajectory set. The evaluation indicators include trajectory length, energy consumption, smoothness, and task completion. The multi-objective optimization uses the NSGA-II (Non-dominated Sorting Genetic Algorithm II) algorithm to search the Pareto frontier through a genetic algorithm to obtain a set of optimal solutions that balance different objectives. Finally, a dynamic constraint analysis is performed on the Pareto optimal trajectory set and the current state of the robot to obtain the target execution trajectory. This step takes into account the maximum joint torque, velocity, and acceleration limits of the robot to ensure that the generated trajectory is physically executable. The dynamic constraint analysis uses inverse dynamics calculations to calculate the required joint torque for each trajectory point and check whether it exceeds the limit.

[0064] For example, in a task where a robot arm needs to pick up a precision part from a workbench and insert it into a product on an assembly line, the task is first decomposed into five basic action units: move above the part, adjust the grasping posture, grasp the part, move above the target product, and perform the insertion operation. Each action unit is represented symbolically, for example, the grasping operation is represented as GRASP (Part 1), the preconditions include HandAbove (Part 1) and HandEmpty, and the post-effect is Holding (Part 1). The 3D reconstruction of the environment obtains a point cloud data containing 500,000 points with a resolution of 1mm. Probabilistic occupancy analysis divides the workspace into a 100x100x100 grid with each voxel size of 1cm³. Dynamic obstacle prediction shows that a human hand will enter the work area in 0.5 seconds, and the occupancy probability increases from 0 to 0.8.

[0065] The predicted state sequence of the robot arm contains 100 time steps, each step is 10ms. The predicted trajectory of the end effector forms a smooth curve in Cartesian space with a length of about 50cm. This trajectory is projected onto the image plane with a resolution of 640x480 to obtain a series of two-dimensional coordinate points. The RRT algorithm generates 10,000 sampling points in the configuration space and obtains 50 candidate trajectories. The multi-objective evaluation considers trajectory length (weight 0.3), energy consumption (weight 0.3), smoothness (weight 0.2) and task completion (weight 0.2). After 100 generations of evolution, the NSGA-II algorithm finally obtained 10 Pareto optimal trajectories.

[0066] The dynamic constraint analysis showed that among the 10 trajectories, 3 trajectories exceeded the joint speed limit at some points (maximum excess of 5%), and 2 trajectories exceeded the torque limit at some points (maximum excess of 3%). The target execution trajectory finally selected was 52cm in length, with an estimated energy consumption of 120J, a smoothness index of 0.95 (out of 1), and a task completion of 1. This trajectory satisfied the dynamic constraints at all time points, with a maximum joint speed of 80% of the rated speed and a maximum torque of 75% of the rated torque. This trajectory was discretized into 200 path points, each containing 6 joint angle values, forming a 6x200 matrix as the input of the subsequent control algorithm.

[0067] Step S105 , generating a control strategy for the target execution trajectory through an event-triggered distributed coordinated control algorithm, obtaining a real-time joint control instruction sequence, and controlling the robot arm to perform operations through the real-time joint control instruction sequence.

[0068] Specifically, the target execution trajectory is discretized and sampled, and the continuous trajectory is converted into a series of key path points. This process uses an equal time interval sampling method, usually at intervals of 10ms, to discretize the trajectory into a series of time-position pairs. Inverse kinematics calculations are performed on these key path points to obtain the initial joint angle sequence. Inverse kinematics calculations use analytical methods combined with numerical iterations to convert the end effector position in Cartesian space into angle values ​​in joint space. Dynamic compensation is performed on the initial joint angle sequence to obtain a compensated joint angle sequence. Dynamic compensation takes into account the mass distribution, inertial characteristics and gravity effects of the robot arm, and uses the Newton-Euler method to calculate the torque required for each joint. The compensation process adjusts the joint angles through inverse dynamics calculations to meet the dynamic constraints and ensure that the trajectory is physically executable.

[0069] Then, the event trigger conditions are constructed for the compensated joint angle sequence and fused feature vector to obtain the trigger threshold set. The event trigger condition is defined based on the state error and control error. For example, when the joint angle error exceeds the preset threshold (such as 0.1 degrees) or the end effector position error exceeds a certain value (such as 1mm), the control update is triggered. The design of the trigger threshold set takes into account the importance and task requirements of different joints. Usually, the trigger threshold of key joints (such as end joints) is set smaller to ensure higher control accuracy. The trigger threshold set is distributedly configured to obtain independent trigger conditions for each joint of the robot arm. The distributed configuration takes into account the coupling relationship and task characteristics between joints, uses graph theory methods to establish a joint network, and distributes trigger conditions through a consistency protocol. For example, adjacent joints may share certain trigger conditions to achieve coordinated control.

[0070] The soft actor-critic algorithm is used to perform reinforcement learning on the compensated joint angle sequence and independent trigger conditions to obtain the optimized control strategy. The soft actor-critic algorithm is a method based on deep reinforcement learning, which includes an actor network (for generating actions) and a critic network (for evaluating action value). During the learning process, the actor network outputs joint control instructions, and the critic network evaluates the effects of these instructions, and continuously optimizes the control strategy through backpropagation. The learning objectives include minimizing trajectory tracking error, energy consumption, and trigger frequency. The optimized control strategy and trigger threshold set are decomposed at multiple sampling rates to obtain a hierarchical control instruction set for vision, force feedback, and joint control. The multi-sampling rate decomposition takes into account the update frequency of different sensors and control loops. For example, visual feedback may be updated at 30Hz, while joint control may run at 1kHz. The hierarchical control instructions ensure control consistency at different time scales through interpolation and prediction methods.

[0071] Next, the fault mode analysis of the hierarchical control instruction set and the independent trigger conditions of each joint is performed to obtain the fault-tolerant control strategy library. The fault mode analysis takes into account sensor failure, actuator abnormality, and other situations, and designs alternative control strategies for each possible fault situation. The fault-tolerant control strategy library is organized using a decision tree structure, which can quickly switch to the appropriate control strategy based on the real-time monitored system status. The human-computer interaction interface mapping of the fault-tolerant control strategy library and the optimized control strategy is performed to obtain a visual control panel. The visual control panel displays the current control status, trigger conditions, and execution trajectory in an intuitive graphical interface, allowing operators to monitor and intervene in the control process in real time.

[0072] Finally, the visual control panel and the hierarchical control instruction set are processed in real time to obtain a real-time joint control instruction sequence. Real-time data stream processing uses a streaming computing framework, such as Apache Flink, to process high-frequency sensor data and control instructions to ensure that control delays are minimized. The generated real-time joint control instruction sequence is directly sent to each joint actuator of the robotic arm to achieve precise control of the robotic arm.

[0073] For example, in a scenario where a 6-DOF robot performs precision assembly tasks, the target execution trajectory is discretized into 500 key path points with a time interval of 10ms. The initial joint angle sequence calculated by inverse kinematics forms a 6x500 matrix. After dynamic compensation, some joint angle values ​​are adjusted by ±0.5 degrees to compensate for the impact of dynamic loads.

[0074] The event trigger conditions are set as follows: the end effector position error threshold is 0.5 mm, the posture error threshold is 0.1 degrees, and the joint angle error threshold is between 0.05 and 0.1 degrees (depending on the importance of the joint). After the distributed configuration, the trigger conditions of the joints close to the end are more stringent, such as the angle error threshold of the sixth joint is set to 0.05 degrees, while the base joint is set to 0.1 degrees.

[0075] The soft actor-critic algorithm uses a three-layer fully connected network as the actor, with the current state (18 dimensions, including the position, velocity and torque of 6 joints) as the input and the control instructions of 6 joints as the output. The critic network structure is similar, but the output is a scalar value indicating the value of the action. After 10,000 iterations of training, the control strategy converged and the average trajectory tracking error was reduced from the initial 2mm to 0.3mm.

[0076] Multi-sampling rate decomposition divides control commands into three layers: visual feedback commands (30Hz), force feedback commands (100Hz) and joint control commands (1kHz). High-frequency control commands are generated from low-frequency commands through interpolation methods to ensure smooth transition.

[0077] The failure mode analysis considers five common failure scenarios, including failure of a single joint encoder, abnormal force sensor, etc. Alternative control strategies are designed for each failure scenario, such as switching to IMU-based attitude estimation when a joint encoder fails.

[0078] Real-time data stream processing uses a 5ms sliding window to process data streams from 100 sensor nodes. Under normal operating conditions, the event trigger frequency is about 50Hz, which greatly reduces the communication burden. The joint control instructions finally generated are updated at a frequency of 1kHz, and the control accuracy reaches ±0.02 degrees, allowing the robot arm to complete precision assembly tasks with sub-millimeter accuracy, such as accurately inserting a 5mm diameter bearing into a 5.01mm diameter hole.

[0079] In the embodiments of the present application, by designing an efficient multimodal data acquisition and preprocessing process, real-time synchronization and noise reduction of visual, force-tactile, joint encoder and inertial measurement unit data are achieved, which greatly improves data quality and processing efficiency, and lays a solid foundation for subsequent feature extraction and fusion; innovative spatial pyramid pooling and adaptive feature selection technology are adopted, combined with wavelet packet transform and principal component analysis, to achieve effective feature extraction of multimodal data, significantly enhancing the system's perception of the environment and tasks; by introducing cross-modal attention networks and multi-level cascade fusion mechanisms, this method successfully solves the problem of heterogeneous data fusion, so that information of different modalities can be organically combined to give full play to their respective advantages; the state estimation and prediction module based on the unscented Kalman filter and recursive neural network greatly improves the system's current state and future trends of the robotic arm. The perception accuracy provides a reliable basis for intelligent decision-making and control; by comprehensively considering the current state, predicted state and environmental constraints of the robot arm, combined with the fast random sampling tree algorithm and multi-objective optimization technology, this method can generate the optimal execution trajectory that not only meets the task requirements but also effectively avoids obstacles, significantly improving the motion planning ability and environmental adaptability of the robot arm; a distributed coordinated control algorithm based on event triggering is proposed, and by designing adaptive trigger thresholds and independent trigger conditions, the collaborative control of multiple joints is realized, which not only reduces unnecessary communication and computing overhead, but also ensures the real-time and accuracy of control; through the efficient fusion of multimodal data, intelligent trajectory planning and adaptive control strategy generation, the operation accuracy, flexibility and task adaptability of the robot arm in complex dynamic environments are comprehensively improved, and the efficiency and accuracy of the robot arm control based on multimodal drive are improved.

[0080] In a specific embodiment, the process of executing step S101 may specifically include the following steps:

[0081] (1) High-speed video is taken of the working space of the robot arm to obtain real-time image stream and real-time video stream, and motion blur removal is performed on the real-time image stream and real-time video stream to obtain clear image data;

[0082] (2) Scan the working space of the robot arm through the depth camera to obtain three-dimensional point cloud data;

[0083] (3) Perform high-frequency sampling on the force-tactile sensor array of the end effector of the robot arm to obtain pressure distribution data, friction and sliding information, and collect data from the encoders of each joint of the robot arm to obtain joint angle, joint angular velocity and joint angular acceleration;

[0084] (4) Collect data from the inertial measurement unit installed on the robotic arm to obtain the linear acceleration and angular velocity of the robotic arm, and collect environmental data through the temperature sensor and humidity sensor installed on the robotic arm to obtain environmental parameter data;

[0085] (5) De-noising the clear image data, 3D point cloud data, and environmental parameter data to obtain initial visual data;

[0086] (6) De-noising the pressure distribution data, friction force, and sliding information to obtain initial force-tactile data;

[0087] (7) De-noising the joint angle, joint angular velocity and joint angular acceleration to obtain initial joint encoder data, and de-noising the robot arm linear acceleration and robot arm angular velocity to obtain initial inertial measurement unit data;

[0088] (8) merging the initial visual data, the initial force-tactile data, and the initial inertial measurement unit data into denoised multimodal data, and performing outlier detection and elimination on the denoised multimodal data through a median filter to obtain smoothed multimodal data;

[0089] (9) Performing time alignment processing on the smooth multimodal data to obtain a multimodal dataset, wherein the multimodal dataset includes: visual data, force-tactile data, joint encoder data, and inertial measurement unit data.

[0090] Specifically, high-speed video recording of the robot's workspace is performed. A high-frame rate camera (such as a 500fps high-speed camera) is used to capture real-time image streams and video streams, and then a deblurring algorithm based on a convolutional neural network (such as DeblurGAN) is applied to eliminate motion blur. The algorithm learns the mapping relationship from blur to clarity through a generative adversarial network. It takes about 20ms to process each frame of the image and obtain clear image data. At the same time, a structured light or time-of-flight (ToF) depth camera is used to perform a 3D scan of the workspace to obtain point cloud data, which is usually updated at a frequency of 30Hz, and each frame contains about 100,000 3D points.

[0091] For force-tactile data acquisition, the force-tactile sensor array installed at the end effector of the robot arm records pressure distribution, friction and sliding information at a sampling rate of 1kHz. These sensors can detect forces in the range of 0.1N to 50N with an accuracy of 0.01N. At the same time, high-precision encoders at each joint collect joint angle data at a frequency of 500Hz with a resolution of 0.01 degrees, and angular velocity and angular acceleration are calculated by numerical differentiation.

[0092] The inertial measurement unit (IMU) installed at the key parts of the robotic arm records linear acceleration and angular velocity at a sampling rate of 200Hz. A typical MEMS IMU can measure acceleration of ±16g and angular velocity of ±2000° / s. Environmental parameters are collected by temperature sensors and humidity sensors with a low update frequency, usually 1Hz, temperature accuracy of ±0.5°C, and humidity accuracy of ±2%RH. De-noising the collected data is a key step to improve data quality. Clear image data and point cloud data are denoised using bilateral filtering, which can smooth noise while maintaining edge information. Since environmental parameter data changes slowly, a simple moving average filter can be used. These processed data together constitute the initial visual data.

[0093] The wavelet threshold denoising method is used to denoise the force-tactile data. The db4 wavelet is selected and decomposed at level 4. The wavelet coefficients are processed using the soft threshold method, and then the signal is reconstructed to obtain the initial force-tactile data. This method can effectively remove high-frequency noise while retaining the rapid change characteristics of the signal. The Kalman filter is used to denoise the joint encoder data and IMU data. For the joint angle, the state equation is established based on the kinematic relationship between the angle, angular velocity and angular acceleration, and the observation equation directly uses the encoder reading. The Kalman filter of the IMU data is based on the rigid body kinematic model and integrates the accelerometer and gyroscope data. After filtering, the initial joint encoder data and initial IMU data are obtained.

[0094] All the initially processed data (vision, force-tactile, joint encoders, IMU) are combined into denoised multimodal data. A median filter is used to detect and remove outliers from these data. The window size of the median filter is adjusted according to the different data types, usually 3-5 samples. This step can effectively remove sudden noise and outliers to obtain smooth multimodal data. Time alignment is performed on the smoothed multimodal data. Using the dynamic time warping (DTW) algorithm, the data with the highest sampling rate (usually force-tactile data, 1kHz) is used as a benchmark to interpolate and align the other modal data. The DTW algorithm finds the best match between different time series through dynamic programming, and can handle the problems of inconsistent sampling rates and time delays.

[0095] For example, in a scene where a 6-DOF robotic arm performs precision assembly tasks, a high-speed camera captures 1280x720 resolution images at a frame rate of 500fps. There is about 5 pixels of motion blur in the original image. After DeblurGAN processing, the blur is reduced to less than 1 pixel, and it takes 18ms to process each frame. The depth camera collects 640x480 resolution depth maps at a frequency of 30Hz, and each frame contains 307,200 3D points.

[0096] The force-tactile sensor array contains 16 force measuring units, each sampling at a frequency of 1kHz. When performing an insertion operation, the maximum pressure recorded was 20N and the minimum detectable pressure was 0.05N. The joint encoder data showed that when performing this operation, the angle of the end joint varied from -30° to +45°, and the maximum angular velocity reached 60° / s.

[0097] The maximum linear acceleration recorded by the IMU data was 2.5g, and the maximum angular velocity was 180° / s. Environmental parameters showed that the working area temperature was maintained at 23±1°C and the relative humidity was 45±3%RH. Wavelet denoising reduced the noise level of the force-tactile data from ±0.1N to ±0.02N. Kalman filtering reduced the joint angle measurement noise from ±0.05° to ±0.01°, and the IMU acceleration measurement noise from ±0.1m / s² to ±0.02m / s².

[0098] The median filter successfully removed three instantaneous spikes (>40N) in the force data, which may be caused by collision or electrical interference. After time alignment, all data were unified to a sampling rate of 1kHz, and the maximum time deviation was reduced from the original 20ms to less than 0.5ms.

[0099] In a specific embodiment, the process of executing step S102 may specifically include the following steps:

[0100] (1) Perform spatial pyramid pooling on visual data to obtain multi-scale visual feature maps, and perform adaptive feature selection on the multi-scale visual feature maps to obtain key visual feature descriptors;

[0101] (2) Performing wavelet packet transform on the force-tactile data to obtain time-frequency domain feature representation, and performing principal component analysis on the time-frequency domain feature representation to obtain target pressure distribution data and texture features;

[0102] (3) Perform kinematic forward and inverse calculations on the joint encoder data and the inertial measurement unit data to obtain the robot arm posture characteristics, and perform differential dynamics modeling on the robot arm posture characteristics to obtain the robot arm dynamic characteristics;

[0103] (4) Temporally align key visual feature descriptors, target pressure distribution data, texture features, robot arm posture features, and robot arm dynamic features to obtain an aligned feature sequence;

[0104] (5) Perform nonlinear interpolation on the aligned feature sequence to obtain feature data with a uniform sampling rate;

[0105] (6) The feature data of uniform sampling rate are weighted by importance through a cross-modal attention network to obtain a weighted feature representation, and the weighted feature representation is cascaded and fused at multiple levels to obtain a fused feature vector.

[0106] Specifically, the visual data is processed by spatial pyramid pooling, which divides the image into grids of different scales and performs pooling operations within each grid. Specifically, a three-level pyramid structure is adopted, which is divided into 1x1, 2x2 and 4x4 grids, and the maximum pooling operation is performed in each grid. For example, for the original image of 1280x720, a 320x180 feature map is obtained at the 4x4 level, a 640x360 feature map is obtained at the 2x2 level, and the original resolution is maintained at the 1x1 level. This multi-scale feature extraction can capture the global and local features of the image and improve the scale invariance of the features. Adaptive feature selection is performed on the multi-scale visual feature map, and the Squeeze-and-Excitation (SE) module is used. The SE module first performs global average pooling on the feature map to obtain the channel descriptor, then learns the correlation between channels through a two-layer fully connected network, and finally re-weights the original feature map with the learned weights. For example, for a feature map with 256 channels, the SE module may increase the weights of some important channels to 1.5 and reduce the weights of unimportant channels to 0.5, thereby highlighting key visual features.

[0107] For force-tactile data, wavelet packet transform is used for time-frequency analysis. Wavelet packet transform is an extension of wavelet transform, which performs multi-layer decomposition of the signal, decomposing not only the low-frequency part but also the high-frequency part. In this method, 5 layers of decomposition are performed using Daubechies wavelet basis to obtain time-frequency features of 32 frequency bands. Principal component analysis (PCA) is performed on these time-frequency features, and the principal components that explain 95% of the variance are retained. Usually, the original high-dimensional features (such as 512 dimensions) are compressed to lower dimensions (such as 50 dimensions) to obtain compressed pressure distribution and texture feature representation.

[0108] The joint encoder data and inertial measurement unit data are first calculated through kinematic forward and inverse solutions to obtain the posture characteristics of the robot arm. Forward kinematics uses the DH parameter method to calculate the position and posture of the end effector, while inverse kinematics uses an analytical method combined with a numerical iteration method to solve the joint angles. For example, for a 6-DOF robot arm, forward kinematics maps the 6 joint angles to the 6-dimensional posture (3-dimensional position and 3-dimensional posture) of the end effector, and inverse kinematics solves it in reverse. Next, the Lagrange equation is used to establish the dynamic model of the robot arm, and the dynamic characteristics of the robot arm are obtained through differential dynamics modeling, including joint torque, inertia matrix, and Coriolis force.

[0109] In order to align the feature sequences of different modalities, the dynamic time warping (DTW) algorithm is used. DTW finds the best alignment path between two time series through dynamic programming and can handle sequences of different lengths and sampling rates. In this method, the visual feature sequence is used as a benchmark to align the feature sequences of other modalities. The aligned feature sequence is resampled by cubic spline interpolation to obtain feature data with a uniform sampling rate. The highest sampling rate (such as 1kHz) is usually selected as the uniform sampling rate. Finally, the importance of the feature data with a uniform sampling rate is weighted by a cross-modal attention network. The network adopts the Transformer architecture and includes a multi-head self-attention mechanism and a cross-modal attention mechanism. The self-attention mechanism is used to capture the temporal dependencies within a single modality, while the cross-modal attention mechanism is used to learn the correlation between different modalities. For example, in an 8-head attention mechanism, each head may focus on different feature combinations, such as vision-tactile, vision-kinematics, etc. The attention weight is normalized by the softmax function and ranges from 0 to 1.

[0110] The weighted feature representation is fused through multiple levels of cascade to form the final fused feature vector. Cascade fusion includes three levels: early fusion, mid-term fusion, and late fusion. Early fusion directly concatenates low-level features of different modalities; mid-term fusion uses convolutional neural networks to fuse intermediate features; and late fusion uses fully connected layers to fuse high-level features. The final fused feature vector is usually a high-dimensional vector, such as 1024 dimensions, which contains comprehensive information of each modality.

[0111] For example, in a precision assembly task, the robot arm needs to insert a microelectronic component into a circuit board. The visual data is subjected to spatial pyramid pooling to obtain feature maps of three scales: 1280x720x64, 640x360x128, and 320x180x256. The SE module adaptively selects these feature maps, increasing the weights of certain key channels (such as edge detection-related channels) to 1.8, while reducing the weights of background-related channels to 0.3. The force-tactile data is transformed by wavelet packet transform to obtain 32 frequency bands of time-frequency features. After principal component analysis, the first 20 principal components are retained, explaining 98.2% of the variance. The position accuracy of the end effector obtained by kinematic calculation of the joint encoder and IMU data reaches ±0.005mm, and the attitude accuracy is ±0.01 degrees. The error of the joint torque estimation obtained by dynamic modeling is less than 1%.

[0112] The DTW algorithm aligns the feature sequences of different modalities, and the maximum time deviation is reduced from the original 50ms to 1ms. In the cross-modal attention network, at the critical moment of the insertion operation, the attention weight of the force-tactile feature rises to 0.6, the visual feature weight is 0.3, and the kinematic feature weight is 0.1. After multi-level cascade fusion, a 1024-dimensional fused feature vector is finally obtained. This fused feature vector enables the robot arm to comprehensively utilize visual, force-tactile and kinematic information to complete the insertion task of micro-components with an accuracy of 5μm and a force control accuracy of 0.1N, and the insertion success rate is increased from the original 90% to 99.5%.

[0113] In a specific embodiment, the process of executing step S103 may specifically include the following steps:

[0114] (1) Map the fused feature vector into joint space to obtain the initial joint state estimation data, and perform kinematic forward solution calculation on the initial joint state estimation data to obtain the end effector pose estimation data;

[0115] (2) performing object position estimation analysis on the end effector pose estimation data and the force-tactile data to obtain target object position estimation data, wherein the target object position estimation data includes the object center of mass coordinates and object contour data;

[0116] (3) Combining the initial joint state estimation data, the end effector pose estimation data, and the target object position estimation data to obtain the system state vector;

[0117] (4) The system state vector is sampled through unscented transformation to obtain a sigma point set, and nonlinear state propagation is performed on the sigma point set to obtain prior state estimation data;

[0118] (5) Use the fused feature vector to update the prior state estimation data to obtain the current state of the robot arm;

[0119] (6) Identify the dynamic parameters of the current state of the robot arm and obtain an adaptive dynamic model;

[0120] (7) The current state of the robot and the adaptive dynamics model are modeled in time series through a gated recurrent unit network to obtain a state evolution sequence;

[0121] (8) Perform robot arm simulation operation analysis on the state evolution sequence to obtain the predicted state of the robot arm.

[0122] Specifically, a fully connected neural network is used to map the high-dimensional fusion feature vector (such as 1024 dimensions) to the joint space of the robot to obtain the initial joint state estimation data. For a 6-DOF robot, the output of this mapping network is an 18-dimensional vector containing the angles, angular velocities, and angular accelerations of the 6 joints. Subsequently, the DH parameter method is used to perform kinematic forward solution calculations to convert the state of the joint space into the end effector pose in Cartesian space. This process involves a series of matrix transformations, and ultimately obtains 6-dimensional end effector pose estimation data, including 3-D position and 3-D attitude (such as Euler angle or quaternion representation). The end effector pose estimation data and force-tactile data are combined to estimate the position of the target object. This process uses a fusion algorithm based on Kalman filtering to combine visual information and force feedback. Visual information provides the approximate position and contour of the object, while force-tactile data provides the precise position and contact force of the contact point. The center of mass coordinates of the object are estimated by the least squares method, and the object contour is reconstructed using shape prior knowledge and edge detection algorithms. The obtained target object position estimation data usually includes 3D centroid coordinates and a set of points or parameterized surfaces describing the object contour.

[0123] The initial joint state estimation data, the end effector pose estimation data and the target object position estimation data are combined into a comprehensive system state vector. For a 6-DOF robot and a target object, this state vector may be around 30 dimensions, containing the complete state of the robot and the position information of the target object.

[0124] The system state vector is sampled using an unscented transformation to generate a set of sigma points. For an n-dimensional state vector, 2n+1 sigma points are usually generated. Each sigma point is a sampling point in the state space and represents a possible realization of the state distribution. These sigma points are propagated nonlinearly through the system model to obtain prior state estimation data. This process takes into account the dynamic model of the robot, including factors such as joint torque, inertia matrix, and Coriolis force. The prior state estimate is updated using the fused eigenvector as observation data. This step uses the unscented Kalman filter (UKF) algorithm to calculate the Kalman gain and correct the state estimate to obtain the current state of the robot. The advantage of UKF is that it can handle highly nonlinear systems without the need for a linearization process, so it can more accurately estimate the state of the robot.

[0125] The obtained current state of the robot is used to identify dynamic parameters and build an adaptive dynamic model. This process uses the recursive least squares (RLS) algorithm to estimate the robot's mass, inertia, friction coefficient and other parameters in real time. The RLS algorithm can converge quickly and adapt to slow changes in parameters, such as load changes or joint wear. The gated recurrent unit (GRU) network is used to model the current state of the robot and the adaptive dynamic model in time series. GRU is a variant of the recurrent neural network that can effectively capture long-term dependencies and has higher computational efficiency than LSTM. The GRU network receives a state sequence and dynamic parameters within a time window as input and outputs state predictions for several future time steps. This process obtains a state evolution sequence that reflects the trend of the robot's state over time.

[0126] The state evolution sequence is analyzed by simulating the operation of the robot arm. This step uses forward dynamics simulation to calculate the motion trajectory of the robot arm in the future time period based on the predicted state and control input. This analysis takes into account the complete robot arm dynamics model, including joint limits, speed and acceleration constraints, and finally obtains the predicted state of the robot arm.

[0127] For example, in a scenario where a 6-DOF robot performs precision assembly tasks, the fused feature vector is a 1024-dimensional vector. It is mapped to an 18-dimensional initial joint state estimate through a fully connected neural network, including the angles, angular velocities, and angular accelerations of the 6 joints. For example, the estimated state of the first joint may be: angle 30°, angular velocity 5° / s, angular acceleration 1° / s². The kinematic forward solution calculation converts these joint states into the pose of the end effector, obtaining the position (x=500mm, y=300mm, z=200mm) and attitude (roll=10°, pitch=5°, yaw=15°).

[0128] Combine force-tactile data to estimate the position of the target object (such as an electronic component). Assume that the force sensor detects a contact force of 0.5N in the negative direction of the z-axis. Combined with visual information, the coordinates of the object's center of mass are estimated to be (505mm, 305mm, 195mm), and the contour data is represented as a 10mm x 5mm x 2mm rectangular prism.

[0129] The system state vector combines this information into a 30-dimensional vector. The unscented transformation samples this vector to generate 61 sigma points. . Each sigma point is propagated through the dynamics model, for example, considering a time step of 0.1 seconds, the possible states after 0.1 seconds are calculated given the initial state and control inputs. The UKF update step uses the fused eigenvector as observation to correct the state estimate. Suppose the initial estimated end position error is 5mm, which is reduced to 1mm after the update. The dynamics parameter identification process may find that the effective mass of the end of the robot arm has increased from 1kg to 1.1kg due to the grasping of the object.

[0130] The GRU network uses 128 hidden units, with inputs of the state sequence and dynamic parameters of the past 10 time steps (10ms per step), and predicts the state of the next 5 time steps. For example, it is predicted that the end effector will move to the position (510mm, 310mm, 190mm) after 0.05 seconds. Finally, the simulation operation analysis of the robot arm takes into account the predicted state and control strategy, and calculates that in the next 0.5 seconds, the robot arm will complete the precise placement of the object, and the predicted final position error is less than 0.1mm. This high-precision state estimation and prediction enables the robot arm to achieve sub-millimeter operation accuracy in complex assembly tasks.

[0131] In a specific embodiment, the process of executing step S104 may specifically include the following steps:

[0132] (1) Decompose the current state of the robot arm into tasks to obtain a basic action unit sequence, and then symbolize the basic action unit sequence to obtain a task execution plan;

[0133] (2) Perform three-dimensional reconstruction of the visual data in the multimodal dataset to obtain dynamic environment point cloud data;

[0134] (3) Perform probabilistic occupancy analysis on the dynamic environment point cloud data to obtain a spatial obstacle distribution map;

[0135] (4) Perform time series prediction on the spatial obstacle distribution map to obtain dynamic obstacle avoidance constraint data, and perform kinematic forward solution on the predicted state of the robot arm and the task execution plan to obtain the predicted trajectory of the end effector;

[0136] (5) Project the predicted trajectory of the end effector onto the image plane to obtain the image feature predicted trajectory, and perform sampling optimization on the image feature predicted trajectory and dynamic obstacle avoidance constraint data to obtain a candidate trajectory set;

[0137] (6) Perform multi-objective evaluation on the candidate trajectory set and task execution plan to obtain the Pareto optimal trajectory set;

[0138] (7) Perform dynamic constraint analysis on the Pareto optimal trajectory set and the current state of the robot arm to obtain the target execution trajectory.

[0139] Specifically, the current state of the robot arm is task-decomposed, and the complex operation task is decomposed into a series of basic action units. This process uses a hierarchical task planning method to gradually refine high-level task goals into executable basic actions. For example, a task of "picking up an object and placing it" can be decomposed into basic action units such as "moving near the object", "adjusting the posture", "grasping the object", "moving to the target position" and "releasing the object".

[0140] Next, these basic action units are symbolically represented to obtain a high-level task execution plan. The symbolic representation uses predicate logic to represent each action unit as an operator with preconditions and postconditions. For example, the "grasp an object" operation can be represented as: GRASP(Object):

[0141] Precondition:At(HandPosition,ObjectPosition);HandEmpty;

[0142] Effect: Holding(Object); ¬HandEmpty

[0143] In this representation:

[0144] GRASP(Object) is the name of the operation, where Object is the object to be grasped.

[0145] Precondition lists the preconditions required to perform this operation:

[0146] At(HandPosition, ObjectPosition) means that the position of the hand is the same as the object position.

[0147] HandEmpty indicates that the robot hand is currently empty (not grasping other objects).

[0148] Effect describes the result of the operation:

[0149] Holding(Object) indicates that the robot is currently holding the object.

[0150] ¬HandEmpty indicates that the hand is no longer empty (the ¬ sign indicates negation).

[0151] This symbolic representation makes it easier for subsequent planning algorithms to understand and process task requirements.

[0152] Next, the visual data in the multimodal dataset is used for 3D reconstruction to obtain dynamic environment point cloud data. This process usually uses structured light or time-of-flight (ToF) depth cameras, combined with simultaneous localization and mapping (SLAM) technology, to generate a high-precision 3D environment model. For example, a typical workspace may be reconstructed into a point cloud containing 500,000 points, each of which contains (x, y, z) coordinates and RGB color information.

[0153] The obtained dynamic environment point cloud data is subjected to probabilistic occupancy analysis to generate a spatial obstacle distribution map. This step uses the Octree data structure to efficiently represent and update the occupancy probability in three-dimensional space. The workspace may be divided into a 100x100x100 grid, with each voxel size of 1cm³. The occupancy probability of each voxel is continuously updated according to the new sensor data through the Bayesian update rule, ranging from 0 (free) to 1 (occupied).

[0154] In order to handle dynamic environments, the spatial obstacle distribution map is predicted in time series to obtain dynamic obstacle avoidance constraint data. Here, the Kalman filter is used to predict the trajectory of the moving object, usually predicting the state within the next 0.5-1 second. For example, if an object moving at a speed of 0.5m / s is detected, the prediction algorithm will calculate the spatial area that the object may occupy in the next 1 second. The predicted state of the robot and the task execution plan are kinematically solved to obtain the predicted trajectory of the end effector. This step uses the DH parameters of the robot and the predicted joint angle sequence to calculate the position and posture sequence of the end effector in Cartesian space. For a 6-DOF robot, this process involves a series of 4x4 matrix transformations, and finally obtains a 3D spatial curve marked by a timestamp.

[0155] The predicted trajectory of the end effector is projected onto the image plane to obtain the predicted trajectory of the image features. This process uses the camera's intrinsic and extrinsic parameter matrices to convert the trajectory points in 3D space into 2D image coordinates. For example, a 3D point (500mm, 300mm, 200mm) may be projected into image coordinates (320, 240). Rapidly Expanding Random Trees The algorithm performs sampling optimization on the image feature predicted trajectory and dynamic obstacle avoidance constraint data to obtain a set of candidate trajectories. The Rapidly Expanding Random Trees algorithm randomly samples in the configuration space to quickly explore feasible paths. In this method, the sampling strategy of the algorithm is modified to favor the area near the image feature predicted trajectory to improve planning efficiency. For example, in one planning, the algorithm may generate 10,000 sampling points and ultimately obtain 50 feasible candidate trajectories.

[0156] A multi-objective evaluation is performed on the obtained candidate trajectory set and task execution plan to obtain the Pareto optimal trajectory set. The evaluation indicators include trajectory length, energy consumption, smoothness, and task completion. The multi-objective optimization uses the NSGA-II (Non-dominated Sorting Genetic Algorithm II) algorithm to search the Pareto frontier through a genetic algorithm. For example, 10 Pareto optimal trajectories may be obtained, each of which makes different trade-offs between different objectives. A dynamic constraint analysis is performed on the Pareto optimal trajectory set and the current state of the manipulator to obtain the target execution trajectory. This step takes into account the maximum joint torque, velocity, and acceleration limits of the manipulator to ensure that the generated trajectory is physically executable. The dynamic constraint analysis uses inverse dynamics calculations to calculate the required joint torque for each trajectory point and check whether it exceeds the limit.

[0157] For example, in a task where a robot arm needs to pick up a precision part from a workbench and insert it into a product on an assembly line, the task is first decomposed into five basic action units: move above the part, adjust the grasping posture, grasp the part, move above the target product, and perform the insertion operation. Each action unit is represented symbolically, for example, the grasping operation is represented as GRASP (Part 1), the preconditions include HandAbove (Part 1) and HandEmpty, and the post-effect is Holding (Part 1). The 3D reconstruction of the environment obtains a point cloud data containing 500,000 points with a resolution of 1mm. Probabilistic occupancy analysis divides the workspace into a grid of 100x100x100, with each voxel size of 1cm³. Dynamic obstacle prediction shows that a human hand will enter the work area in 0.5 seconds, and the occupancy probability increases from 0 to 0.8. The robot arm predicts that the state sequence contains 100 time steps, each step is 10ms. The end effector predicted trajectory forms a smooth curve in Cartesian space with a length of about 50cm.

[0158] The rapidly expanding random tree algorithm generated 10,000 sampling points in the configuration space and obtained 50 candidate trajectories. The multi-objective evaluation considered trajectory length (weight 0.3), energy consumption (weight 0.3), smoothness (weight 0.2), and task completion (weight 0.2). After 100 generations of evolution, the NSGA-II algorithm finally obtained 10 Pareto optimal trajectories. The dynamic constraint analysis showed that among these 10 trajectories, 3 trajectories exceeded the joint speed limit at some points (maximum excess of 5%), and 2 trajectories exceeded the torque limit at some points (maximum excess of 3%). The target execution trajectory finally selected had a length of 52cm, an estimated energy consumption of 120J, a smoothness index of 0.95 (full score 1), and a task completion of 1. This trajectory satisfied the dynamic constraints at all time points, with a maximum joint speed of 80% of the rated speed and a maximum torque of 75% of the rated torque. This trajectory is discretized into 200 path points, each containing 6 joint angle values, forming a 6x200 matrix as the input of the subsequent control algorithm.

[0159] In a specific embodiment, the process of executing step S105 may specifically include the following steps:

[0160] (1) Discretely sample the target execution trajectory to obtain a key path point sequence, and perform inverse kinematics calculation on the key path point sequence to obtain the initial joint angle sequence;

[0161] (2) Perform dynamic compensation on the initial joint angle sequence to obtain a compensated joint angle sequence;

[0162] (3) Construct event trigger conditions for the compensation joint angle sequence and fusion feature vector to obtain a trigger threshold set, and distribute the trigger threshold set to obtain independent trigger conditions for each joint of the robotic arm;

[0163] (4) Reinforcement learning is performed on the compensation joint angle sequence and the independent trigger conditions of each joint of the robot arm to obtain the optimized control strategy;

[0164] (5) Perform multi-sampling rate decomposition on the optimized control strategy and trigger threshold set to obtain a hierarchical control instruction set for vision, force feedback, and joint control;

[0165] (6) Perform failure mode analysis on the hierarchical control instruction set and the independent trigger conditions of each joint of the robot arm to obtain a fault-tolerant control strategy library;

[0166] (7) Map the fault-tolerant control strategy library and the optimization control strategy to the human-computer interaction interface to obtain a visual control panel;

[0167] (8) Perform real-time data stream processing on the visual control panel and the hierarchical control instruction set to obtain a real-time joint control instruction sequence, and control the robot arm to perform operations through the real-time joint control instruction sequence.

[0168] Specifically, the key step in implementing the multimodal drive-based robotic arm control method begins with the discretization sampling of the target execution trajectory. This process uses an equal time interval sampling method, usually at intervals of 10ms, to convert the continuous trajectory into a series of key path points. For example, a 5-second trajectory may be discretized into 500 path points, each of which contains the position and posture information of the end effector in Cartesian space. Inverse kinematics calculations are performed on these key path points to obtain the initial joint angle sequence. Inverse kinematics calculations use analytical methods combined with numerical iterations, such as the Newton-Raphson method, to convert the end effector position in Cartesian space into angle values ​​in joint space. For a 6-DOF robotic arm, this step will generate a 6x500 matrix, with each column representing 6 joint angles at a time point.

[0169] The initial joint angle sequence is dynamically compensated to obtain the compensated joint angle sequence. Dynamic compensation takes into account the mass distribution, inertial characteristics and gravity effects of the robot arm, and uses the Newton-Euler method to calculate the torque required for each joint. The compensation process adjusts the joint angles to meet the dynamic constraints through inverse dynamics calculations to ensure that the trajectory is physically executable. For example, considering the effects of load and gravity, the angles of some joints may need to be adjusted by ±0.5 degrees. Event trigger conditions are constructed for the compensated joint angle sequence and fused feature vector to obtain a trigger threshold set. Event trigger conditions are defined based on state error and control error, such as triggering a control update when the joint angle error exceeds 0.1 degrees or the end effector position error exceeds 1mm. The design of the trigger threshold set takes into account the importance and task requirements of different joints. Usually, the trigger thresholds of key joints (such as end joints) are set smaller to ensure higher control accuracy.

[0170] The trigger threshold set is distributedly configured to obtain independent trigger conditions for each joint of the robot. The distributed configuration takes into account the coupling relationship and task characteristics between joints, uses graph theory to establish a joint network, and distributes trigger conditions through a consistency protocol. For example, adjacent joints may share certain trigger conditions to achieve coordinated control. The soft actor-critic algorithm is used to perform reinforcement learning on the compensation joint angle sequence and independent trigger conditions to obtain an optimized control strategy. The soft actor-critic algorithm includes an actor network (for generating actions) and a critic network (for evaluating action values). During the learning process, the actor network outputs joint control instructions, and the critic network evaluates the effects of these instructions, and continuously optimizes the control strategy through back propagation. The learning objectives include minimizing trajectory tracking error, energy consumption, and trigger frequency.

[0171] The optimized control strategy and trigger threshold set are decomposed at multiple sampling rates to obtain a hierarchical control instruction set for vision, force feedback, and joint control. The multi-sampling rate decomposition takes into account the update frequency of different sensors and control loops, such as visual feedback updated at 30Hz, force feedback updated at 100Hz, and joint control running at 1kHz. The hierarchical control instructions ensure control consistency on different time scales through interpolation and prediction methods. Fault mode analysis is performed on the hierarchical control instruction set and the independent trigger conditions of each joint to obtain a fault-tolerant control strategy library. The fault mode analysis takes into account sensor failure, actuator abnormality, etc., and designs alternative control strategies for each possible fault situation. The fault-tolerant control strategy library is organized using a decision tree structure, which can quickly switch to the appropriate control strategy based on the real-time monitored system status.

[0172] The fault-tolerant control strategy library and the optimization control strategy are mapped to a human-computer interaction interface to obtain a visual control panel. The visual control panel displays the current control status, trigger conditions, and execution trajectory in an intuitive graphical interface, allowing operators to monitor and intervene in the control process in real time.

[0173] The visual control panel and hierarchical control instruction set are processed in real time to obtain a real-time joint control instruction sequence. Real-time data stream processing uses a streaming computing framework, such as Apache Flink, to process high-frequency sensor data and control instructions to ensure that control delays are minimized. The generated real-time joint control instruction sequence is directly sent to each joint actuator of the robotic arm to achieve precise control of the robotic arm.

[0174] For example, in a scenario where a 6-DOF robot performs a precision assembly task, the target execution trajectory is discretized into 500 key path points with a time interval of 10ms. The initial joint angle sequence obtained by inverse kinematics calculation forms a 6x500 matrix. After dynamic compensation, some joint angle values ​​are adjusted by ±0.5 degrees to compensate for the influence of dynamic loads. The event trigger conditions are set as follows: the end effector position error threshold is 0.5mm, the posture error threshold is 0.1 degrees, and the joint angle error threshold is between 0.05-0.1 degrees (depending on the importance of the joint). After distributed configuration, the trigger conditions of the joints close to the end are stricter, such as the angle error threshold of the 6th joint is set to 0.05 degrees, while the base joint is set to 0.1 degrees. The soft actor-critic algorithm uses a three-layer fully connected network as the actor, with the current state (18 dimensions, including the position, velocity and torque of 6 joints) as the input and the control instructions of 6 joints as the output. The critic network structure is similar, but the output is a scalar value indicating the value of the action. After 10,000 iterations of training, the control strategy converged and the average trajectory tracking error was reduced from the initial 2mm to 0.3mm.

[0175] Multi-sampling rate decomposition divides the control commands into three layers: visual feedback commands (30Hz), force feedback commands (100Hz), and joint control commands (1kHz). High-frequency control commands are generated from low-frequency commands through interpolation methods to ensure smooth transition. Failure mode analysis considers five common failure scenarios, including failure of a single joint encoder and abnormal force sensor. Alternative control strategies are designed for each failure scenario, such as switching to IMU-based attitude estimation when a joint encoder fails.

[0176] Real-time data stream processing uses a 5ms sliding window to process data streams from 100 sensor nodes. Under normal operating conditions, the event trigger frequency is about 50Hz, which greatly reduces the communication burden. The joint control instructions finally generated are updated at a frequency of 1kHz, and the control accuracy reaches ±0.02 degrees, allowing the robot arm to complete precision assembly tasks with sub-millimeter accuracy, such as accurately inserting a 5mm diameter bearing into a 5.01mm diameter hole, and the maximum force error during the insertion process is controlled within 0.1N.

[0177] In a specific embodiment, the process of constructing event trigger conditions for the compensation joint angle sequence and the fusion feature vector to obtain a trigger threshold set, and distributively configuring the trigger threshold set to obtain independent trigger conditions for each joint of the robotic arm may specifically include the following steps:

[0178] (1) Perform joint space analysis on the compensation joint angle sequence to obtain the joint motion range;

[0179] (2) Calculate the torque threshold of the force-tactile data in the multimodal dataset to obtain the joint torque range, and perform constraint analysis on the joint motion range and joint torque range to obtain the preliminary trigger conditions;

[0180] (3) Estimate the position of the target object based on the visual data in the multimodal dataset to obtain the end effector position error, and perform Jacobian matrix mapping on the end effector position error to obtain the joint angle error threshold;

[0181] (4) Perform weighted fusion of the preliminary trigger condition and the joint angle error threshold to obtain the comprehensive trigger threshold;

[0182] (5) Calculate the dynamic adjustment coefficient of the comprehensive trigger threshold to obtain an adaptive trigger threshold set;

[0183] (6) Perform noise sensitivity analysis on the adaptive trigger threshold set to obtain the anti-interference trigger threshold;

[0184] (7) Perform joint coupling analysis on the anti-interference trigger threshold to obtain a trigger threshold set, and then map the trigger threshold set to the robot topology structure to obtain the independent trigger condition of each joint of the robot.

[0185] Specifically, the range of motion of each joint during the entire task execution is calculated by statistical methods, including the maximum angle, minimum angle, and angular velocity limits. For example, for a 6-DOF robotic arm, the range of motion of the first joint (base rotation) may be [-180°, 180°], while the range of motion of the end joint may be smaller, such as [-90°, 90°]. The torque threshold is calculated for the force-tactile data in the multimodal dataset to obtain the joint torque range. This process uses the data from the force-tactile sensor combined with the dynamic model of the robotic arm to calculate the maximum torque that each joint may encounter when performing the task. For example, when the end effector grasps a 10kg object, it may cause the second joint (shoulder) to be subjected to a torque of 100Nm.

[0186] The joint motion range and joint torque range are constrained and analyzed to obtain preliminary trigger conditions. This step takes into account the kinematic and dynamic constraints and sets preliminary trigger thresholds for each joint. Generally, joints with torques close to the limit values ​​are given more sensitive trigger conditions. For example, if the maximum allowable torque of a joint is 150Nm, a control update may be triggered when the actual torque exceeds 120Nm (80% threshold). The target object position is estimated for the visual data in the multimodal dataset to obtain the end effector position error. This process uses computer vision algorithms, such as deep learning-based target detection and posture estimation methods, to calculate the relative position between the target object and the end effector. For example, in a precision assembly task, if the position deviation between the end effector and the target socket is detected to exceed 0.5mm, a control update needs to be triggered.

[0187] The Jacobian matrix is ​​mapped to the end effector position error to obtain the joint angle error threshold. The Jacobian matrix describes the sensitivity of the end effector position to the joint angle. The end position error can be converted into the angle error of each joint through the inverse operation of the Jacobian matrix. For example, an end position error of 0.5mm may correspond to an angle error of 0.1° in the sixth joint, but only 0.01° in the first joint. The preliminary trigger condition and the joint angle error threshold are weighted and fused to obtain the comprehensive trigger threshold. This step uses weighted average or fuzzy logic methods to comprehensively consider the torque constraints and accuracy requirements. The distribution of weights reflects the characteristics of the task. For example, in force control tasks, the weight of the torque threshold may be higher.

[0188] The dynamic adjustment coefficient of the comprehensive trigger threshold is calculated to obtain the adaptive trigger threshold set. This process takes into account the different stages of the task and environmental changes, and dynamically adjusts the trigger sensitivity. For example, when approaching a target object, the trigger threshold may be automatically reduced by 20% to improve control accuracy. The noise sensitivity analysis of the adaptive trigger threshold set is performed to obtain the anti-interference trigger threshold. This step uses Monte Carlo simulation or statistical analysis methods to evaluate the reliability of the trigger conditions under different noise levels. For example, if the standard deviation of the sensor noise is 0.01°, the trigger threshold may need to be set to 0.03° (3 times the standard deviation) to avoid false triggers.

[0189] Then, the joint coupling analysis of the anti-interference trigger threshold is performed to obtain the final set of trigger thresholds. This process takes into account the dynamic coupling between the joints of the manipulator and uses structured learning or graph model methods to establish a relationship network between the joints. For example, the trigger conditions of adjacent joints may need to be set collaboratively to prevent oscillation. The trigger threshold set is mapped to the manipulator topology to obtain independent trigger conditions for each joint of the manipulator. This step assigns the globally optimized trigger thresholds to each joint, taking into account the geometric structure and motion characteristics of the manipulator. For example, the joints close to the base may have larger trigger thresholds, while the thresholds of the end joints are more stringent.

[0190] For example, in a scenario where a 6-DOF robot performs a precision insertion task, the range of motion of each joint is first obtained through joint space analysis: joint 1 is [-180°, 180°], joint 2 is [-90°, 90°], joint 3 is [-120°, 120°], joint 4 is [-180°, 180°], joint 5 is [-90°, 90°], and joint 6 is [-360°, 360°]. The torque threshold calculation shows that during the insertion process, the second joint may be subjected to a maximum torque of 100Nm, while the torques of other joints are smaller. The vision system detects the target socket position and calculates the position error of the end effector to be 0.3mm. Through Jacobian matrix mapping, this error is converted into joint angle errors: joint 1 is 0.01°, joint 2 is 0.02°, joint 3 is 0.03°, joint 4 is 0.05°, joint 5 is 0.08°, and joint 6 is 0.1°.

[0191] Considering the torque and accuracy requirements, the initial trigger condition is set as follows: triggering when the torque exceeds 80% of the maximum allowable value or the angle error exceeds 2 times the above value. During dynamic adjustment, all thresholds are reduced by 25% within the last 1cm of insertion to improve accuracy. Noise analysis shows that the standard deviation of sensor noise is 0.005°, so the trigger threshold is set to 3 times the standard deviation, that is, 0.015°. Considering the coupling of joints, the trigger conditions of adjacent joints are set collaboratively. For example, the triggering of joints 2 and 3 needs to meet their respective conditions at the same time. Finally, after mapping to the robot arm topology, the independent trigger conditions of each joint are obtained: 0.03° or 80Nm for joint 1, 0.04° or 90Nm for joint 2, 0.06° or 70Nm for joint 3, 0.1° or 50Nm for joint 4, 0.16° or 30Nm for joint 5, and 0.2° or 20Nm for joint 6. These trigger conditions ensure that the robot arm can respond to changes in position and force in a timely manner when performing precision insertion tasks, achieving sub-millimeter operating accuracy.

[0192] The above describes the robot arm control method based on multi-modal drive in the embodiment of the present application. The following describes the robot arm control system based on multi-modal drive in the embodiment of the present application. Figure 2 In the embodiment of the present application, an embodiment of a robotic arm control system based on multi-modal drive includes:

[0193] The processing module 201 is used to collect data through the sensor module on the robot arm and perform preprocessing to obtain a multimodal data set, wherein the multimodal data set includes: visual data, force-tactile data, joint encoder data and inertial measurement unit data;

[0194] A fusion module 202 is used to extract features from the multimodal data set and fuse the features through an attention mechanism to obtain a fused feature vector;

[0195] A prediction module 203 is used to estimate the state data of the fused feature vector through an unscented Kalman filter and perform state prediction through a recursive neural network to obtain the current state of the robot arm and the predicted state of the robot arm;

[0196] An analysis module 204 is used to perform a robot arm operation trajectory planning analysis on the current state of the robot arm and the predicted state of the robot arm to obtain a target execution trajectory;

[0197] The generation module 205 is used to generate a control strategy for the target execution trajectory through an event-triggered distributed coordinated control algorithm, obtain a real-time joint control instruction sequence, and control the robot arm to perform operations through the real-time joint control instruction sequence.

[0198] Through the collaborative cooperation of the above-mentioned components and the design of efficient multimodal data acquisition and preprocessing processes, real-time synchronization and noise reduction of visual, force-tactile, joint encoder and inertial measurement unit data are achieved, which greatly improves data quality and processing efficiency, and lays a solid foundation for subsequent feature extraction and fusion; innovative spatial pyramid pooling and adaptive feature selection techniques are adopted, combined with wavelet packet transform and principal component analysis, to achieve effective feature extraction of multimodal data, significantly enhancing the system's perception of the environment and tasks; by introducing cross-modal attention networks and multi-level cascade fusion mechanisms, this method successfully solves the problem of heterogeneous data fusion, so that information from different modalities can be organically combined to give full play to their respective advantages; based on the state estimation and prediction module of the unscented Kalman filter and recursive neural network, the system's current and future state of the robot arm are greatly improved. The perception accuracy of future trends provides a reliable basis for intelligent decision-making and control; by comprehensively considering the current state, predicted state and environmental constraints of the robot arm, combined with the fast random sampling tree algorithm and multi-objective optimization technology, this method can generate the optimal execution trajectory that not only meets the task requirements but also effectively avoids obstacles, significantly improving the motion planning ability and environmental adaptability of the robot arm; a distributed coordinated control algorithm based on event triggering is proposed, and by designing adaptive trigger thresholds and independent trigger conditions, the collaborative control of multiple joints is realized, which not only reduces unnecessary communication and computing overhead, but also ensures the real-time and accuracy of control; through the efficient fusion of multimodal data, intelligent trajectory planning and adaptive control strategy generation, the operation accuracy, flexibility and task adaptability of the robot arm in complex dynamic environments are comprehensively improved, and the efficiency and accuracy of the robot arm control based on multimodal drive are improved.

[0199] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the multi-modal drive-based robotic arm control method.

[0200] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0201] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A robotic arm control method based on multi-modal drive, characterized in that: The multi-mode drive-based robotic arm control method comprises: The sensor module on the robot arm collects data and performs preprocessing to obtain a multimodal data set, wherein the multimodal data set includes: visual data, force-tactile data, joint encoder data, and inertial measurement unit data; Extract features from the multimodal dataset and fuse the features through an attention mechanism to obtain a fused feature vector; The state data of the fused feature vector is estimated by an unscented Kalman filter, and the state is predicted by a recursive neural network to obtain the current state of the robot arm and the predicted state of the robot arm; Performing robot arm operation trajectory planning analysis on the current state of the robot arm and the predicted state of the robot arm to obtain a target execution trajectory; A control strategy is generated for the target execution trajectory through an event-triggered distributed coordinated control algorithm to obtain a real-time joint control instruction sequence, and the robot arm is controlled to perform operations through the real-time joint control instruction sequence; The distributed coordinated control algorithm triggered by events generates a control strategy for the target execution trajectory to obtain a real-time joint control instruction sequence, and controls the robot arm to perform operations through the real-time joint control instruction sequence, including: discretizing the target execution trajectory to obtain a key path point sequence, and performing inverse kinematics calculation on the key path point sequence to obtain an initial joint angle sequence; performing dynamic compensation on the initial joint angle sequence to obtain a compensated joint angle sequence; constructing event trigger conditions for the compensated joint angle sequence and the fused feature vector to obtain a trigger threshold set, and distributively configuring the trigger threshold set to obtain independent trigger conditions for each joint of the robot arm; and performing inverse kinematics calculation on the compensated joint angle sequence to obtain an independent trigger condition for each joint of the robot arm. The joint angle sequence and the independent trigger conditions of each joint of the robotic arm are subjected to reinforcement learning to obtain an optimized control strategy; the optimized control strategy and the trigger threshold set are subjected to multi-sampling rate decomposition to obtain a hierarchical control instruction set of vision, force feedback and joint control; the hierarchical control instruction set and the independent trigger conditions of each joint of the robotic arm are subjected to fault mode analysis to obtain a fault-tolerant control strategy library; the fault-tolerant control strategy library and the optimized control strategy are subjected to human-computer interaction interface mapping to obtain a visual control panel; the visual control panel and the hierarchical control instruction set are subjected to real-time data stream processing to obtain the real-time joint control instruction sequence, and the robotic arm is controlled to perform operations through the real-time joint control instruction sequence.

2. The method for controlling a robotic arm based on multi-modal drive according to claim 1, characterized in that: The sensor module on the robot arm collects data and performs preprocessing to obtain a multimodal data set, including: Performing high-speed video recording of the working space of the robot arm to obtain a real-time image stream and a real-time video stream, and performing motion blur elimination processing on the real-time image stream and the real-time video stream to obtain clear image data; Scanning the working space of the robotic arm by a depth camera to obtain three-dimensional point cloud data; Performing high-frequency sampling on the force-tactile sensor array of the end effector of the robotic arm to obtain pressure distribution data, friction force and sliding information, and collecting data on the encoders of each joint of the robotic arm to obtain joint angles, joint angular velocities and joint angular accelerations; Collecting data from an inertial measurement unit installed on the robotic arm to obtain the linear acceleration and angular velocity of the robotic arm, and collecting environmental data through a temperature sensor and a humidity sensor installed on the robotic arm to obtain environmental parameter data; De-noising the clear image data, the three-dimensional point cloud data, and the environmental parameter data to obtain initial visual data; De-noising the pressure distribution data, the friction force and the sliding information to obtain initial force-tactile data; De-noising the joint angle, the joint angular velocity and the joint angular acceleration to obtain initial joint encoder data, and de-noising the mechanical arm linear acceleration and the mechanical arm angular velocity to obtain initial inertial measurement unit data; Combining the initial visual data, the initial force-tactile data, and the initial inertial measurement unit data into denoised multimodal data, and performing outlier detection and elimination on the denoised multimodal data through a median filter to obtain smoothed multimodal data; The smoothed multimodal data is subjected to time alignment processing to obtain the multimodal data set, wherein the multimodal data set includes: visual data, force-tactile data, joint encoder data and inertial measurement unit data.

3. The method for controlling a robotic arm based on multi-modal drive according to claim 2, characterized in that: The extracting features of the multimodal dataset and fusing the features through an attention mechanism to obtain a fused feature vector includes: Performing spatial pyramid pooling processing on the visual data to obtain a multi-scale visual feature map, and performing adaptive feature selection on the multi-scale visual feature map to obtain a key visual feature descriptor; Performing wavelet packet transformation on the force-tactile data to obtain time-frequency domain feature representation, and performing principal component analysis on the time-frequency domain feature representation to obtain target pressure distribution data and texture features; Perform kinematic forward and inverse calculations on the joint encoder data and the inertial measurement unit data to obtain the posture characteristics of the robotic arm, and perform differential dynamics modeling on the posture characteristics of the robotic arm to obtain the dynamic characteristics of the robotic arm; Performing time alignment on the key visual feature descriptor, the target pressure distribution data, the texture feature, the robotic arm posture feature, and the robotic arm dynamic feature to obtain an aligned feature sequence; Nonlinear interpolation is performed on the aligned feature sequence to obtain feature data of a uniform sampling rate; importance is weighted on the feature data of the uniform sampling rate through a cross-modal attention network to obtain a weighted feature representation; and multi-level cascade fusion is performed on the weighted feature representation to obtain the fused feature vector.

4. The method for controlling a robotic arm based on multi-modal drive according to claim 1, characterized in that: The state data of the fused feature vector is estimated by using an unscented Kalman filter, and the state is predicted by using a recursive neural network to obtain the current state of the robot arm and the predicted state of the robot arm, including: Performing joint space mapping on the fused feature vector to obtain initial joint state estimation data, and performing kinematic forward solution calculation on the initial joint state estimation data to obtain end effector position and posture estimation data; Performing object position estimation analysis on the end effector posture estimation data and the force-tactile data to obtain target object position estimation data, wherein the target object position estimation data includes object center of mass coordinates and object contour data; Combining the initial joint state estimation data, the end effector posture estimation data and the target object position estimation data to obtain a system state vector; The system state vector is sampled by untraceable transformation to obtain a sigma point set, and the sigma point set is subjected to nonlinear state propagation to obtain prior state estimation data; the prior state estimation data is updated by using the fused feature vector to obtain the current state of the robot arm; Performing dynamic parameter identification on the current state of the robotic arm to obtain an adaptive dynamic model; Performing time series modeling on the current state of the robot and the adaptive dynamics model through a gated recurrent unit network to obtain a state evolution sequence; A robot arm simulation operation analysis is performed on the state evolution sequence to obtain the predicted state of the robot arm.

5. The method for controlling a robotic arm based on multi-modal driving according to claim 3, characterized in that: The step of performing robot arm operation trajectory planning analysis on the current state of the robot arm and the predicted state of the robot arm to obtain a target execution trajectory includes: Decomposing the current state of the robot arm into tasks to obtain a basic action unit sequence, and symbolically representing the basic action unit sequence to obtain a task execution plan; Performing three-dimensional reconstruction on the visual data in the multimodal data set to obtain dynamic environment point cloud data; Performing probability occupancy analysis on the dynamic environment point cloud data to obtain a spatial obstacle distribution map; Performing time series prediction on the spatial obstacle distribution map to obtain dynamic obstacle avoidance constraint data, and performing kinematic forward solution on the predicted state of the robotic arm and the task execution plan to obtain a predicted trajectory of the end effector; Performing image plane projection on the predicted trajectory of the end effector to obtain an image feature predicted trajectory, and performing sampling optimization on the image feature predicted trajectory and the dynamic obstacle avoidance constraint data to obtain a candidate trajectory set; Performing a multi-objective evaluation on the candidate trajectory set and the task execution plan to obtain a Pareto optimal trajectory set; A dynamic constraint analysis is performed on the Pareto optimal trajectory set and the current state of the robotic arm to obtain the target execution trajectory.

6. The method for controlling a robotic arm based on multi-modal driving according to claim 1, characterized in that: The constructing of event trigger conditions for the compensation joint angle sequence and the fusion feature vector to obtain a trigger threshold set, and distributing the trigger threshold set to obtain an independent trigger condition for each joint of the robotic arm includes: Performing joint space analysis on the compensation joint angle sequence to obtain the joint motion range; performing torque threshold calculation on the force-tactile data in the multimodal data set to obtain the joint torque range, and performing constraint analysis on the joint motion range and the joint torque range to obtain preliminary trigger conditions; Estimating the position of the target object on the visual data in the multimodal data set to obtain an end effector position error, and performing Jacobian matrix mapping on the end effector position error to obtain a joint angle error threshold; Performing weighted fusion on the preliminary trigger condition and the joint angle error threshold to obtain a comprehensive trigger threshold; Calculating a dynamic adjustment coefficient of the comprehensive trigger threshold to obtain an adaptive trigger threshold set; Performing noise sensitivity analysis on the adaptive trigger threshold set to obtain an anti-interference trigger threshold; A joint coupling analysis is performed on the anti-interference trigger threshold to obtain the trigger threshold set, and a robotic arm topology mapping is performed on the trigger threshold set to obtain an independent trigger condition for each joint of the robotic arm.

7. A robotic arm control system based on multi-modal drive, characterized in that: Used to execute the multi-modal drive-based robotic arm control method according to any one of claims 1 to 6, the multi-modal drive-based robotic arm control system comprising: A processing module, used for collecting data through a sensor module on the robotic arm and performing preprocessing to obtain a multimodal data set, wherein the multimodal data set includes: visual data, force-tactile data, joint encoder data and inertial measurement unit data; A fusion module, used to extract features from the multimodal dataset and fuse the features through an attention mechanism to obtain a fused feature vector; A prediction module is used to estimate the state data of the fused feature vector through an unscented Kalman filter and to predict the state through a recursive neural network to obtain the current state of the robot arm and the predicted state of the robot arm; An analysis module is used to perform robot arm operation trajectory planning analysis on the current state of the robot arm and the predicted state of the robot arm to obtain a target execution trajectory; A generation module is used to generate a control strategy for the target execution trajectory through an event-triggered distributed coordination control algorithm to obtain a real-time joint control instruction sequence, and control the robotic arm to perform operations through the real-time joint control instruction sequence. The control strategy is generated for the target execution trajectory through an event-triggered distributed coordination control algorithm to obtain a real-time joint control instruction sequence, and the robotic arm is controlled to perform operations through the real-time joint control instruction sequence, including: discretizing the target execution trajectory to obtain a key path point sequence, and performing inverse kinematics calculation on the key path point sequence to obtain an initial joint angle sequence; performing dynamic compensation on the initial joint angle sequence to obtain a compensated joint angle sequence; constructing event trigger conditions for the compensated joint angle sequence and the fusion feature vector to obtain a trigger threshold set, and The trigger threshold set is distributedly configured to obtain independent trigger conditions for each joint of the robotic arm; reinforcement learning is performed on the compensation joint angle sequence and the independent trigger conditions of each joint of the robotic arm to obtain an optimized control strategy; multi-sampling rate decomposition is performed on the optimized control strategy and the trigger threshold set to obtain a hierarchical control instruction set of vision, force feedback and joint control; fault mode analysis is performed on the hierarchical control instruction set and the independent trigger conditions of each joint of the robotic arm to obtain a fault-tolerant control strategy library; human-computer interaction interface mapping is performed on the fault-tolerant control strategy library and the optimized control strategy to obtain a visual control panel; real-time data stream processing is performed on the visual control panel and the hierarchical control instruction set to obtain the real-time joint control instruction sequence, and the robotic arm is controlled to perform operations through the real-time joint control instruction sequence.

8. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the robotic arm control method based on multi-modal drive as described in any one of claims 1 to 6 is implemented.

Images