A Method and System for Dynamic Gravity Compensation Control of Robotic Arms Based on Multimodal Prediction

Abstract

This application relates to a method and system for dynamic gravity compensation control of a robotic arm based on multimodal prediction. The method includes: collecting load information data, including joint torque values, end-effector acceleration values, and joint angle values, within the current operating cycle of the robotic arm using multimodal sensors; preprocessing this data to generate a standardized dataset; using a trained deep learning model to predict the dynamic characteristics of the load in the next operating cycle, accurately grasping the load mass distribution and center of gravity position change trends; calculating the gravity compensation torque value based on the prediction results using a robotic arm dynamics model, making the compensation more accurate; and feeding the gravity compensation torque value back to the control system in real time to adjust the output torque of the drive motor, thereby achieving dynamic gravity compensation control of the robotic arm. This scheme can effectively improve the motion accuracy of the robotic arm during operation, reduce positioning deviations and operational errors caused by gravity, and ensure stable and efficient operation of the robotic arm in complex load environments.

Description

Technical Field

[0001] This application relates to the fields of intelligent control and robotics, and in particular to a method and system for dynamic gravity compensation control of a robotic arm based on multimodal prediction. Background Technology

[0002] In the field of industrial automation, robotic arms, as a key type of automation equipment, are widely used in many production processes such as material handling, parts assembly, and welding. During operation, the load on a robotic arm is complex and variable, and the load's gravity significantly affects its motion accuracy and stability. Accurately compensating for and controlling the weight of the robotic arm is one of the core technologies for ensuring its efficient and precise operation.

[0003] Currently, some traditional methods involve theoretical analysis of the structure and load of the robotic arm during the design phase, calculating gravity compensation values ​​based on a fixed robotic arm dynamics model. However, in actual production scenarios, the load carried by the robotic arm often exhibits uncertainty and dynamic variability. For example, on an assembly line, the weight and center of gravity of parts from different batches may vary, and the posture of the load on the robotic arm may also change during movement.

[0004] These traditional gravity compensation methods, based on fixed models, cannot adapt to dynamic changes in load in real time. When the dynamic changes in load exceed the model's preset range, the accuracy of gravity compensation drops significantly, causing deviations in the robotic arm's movement. This, in turn, affects production efficiency and product quality, failing to meet the high precision and high stability requirements of modern industrial automation for robotic arms. Summary of the Invention

[0005] The main objective of this application is to provide a multimodal prediction-based dynamic gravity compensation control method and system for robotic arms, which can improve the accuracy of gravity compensation for robotic arms, thereby improving the motion precision of robotic arms.

[0006] To achieve the above objectives, embodiments of the present invention provide a method for dynamic gravity compensation control of a robotic arm based on multimodal prediction, the method comprising:

[0007] The load information data collected by the robotic arm through multimodal sensors during the current operating cycle is acquired. The load information data includes the joint torque value recorded by the torque sensor, the end-effector acceleration value recorded by the acceleration sensor, and the joint angle value recorded by the angle sensor.

[0008] The load information data is preprocessed to generate a standardized dataset corresponding to the operating state of the robotic arm. The standardized dataset includes torque change curves, acceleration change curves, and angle change curves in a time series.

[0009] The standardized dataset is input into the trained deep learning model, and the load dynamic characteristics of the robotic arm in the next operating cycle are predicted based on the deep learning model. The load dynamic characteristics include load mass distribution characteristics and load center of gravity position change characteristics.

[0010] Based on the predicted results of the load dynamic characteristics, the gravity compensation torque value of each joint of the robotic arm in the next operating cycle is calculated. The gravity compensation torque value is obtained by combining the robotic arm dynamic model with the load dynamic characteristics.

[0011] The gravity compensation torque value is fed back to the control system of the robotic arm in real time, and the output torque of the drive motor of each joint of the robotic arm is adjusted to realize the dynamic gravity compensation control of the robotic arm in the next operating cycle.

[0012] In summary, the technical solution of this application collects load information data of the robotic arm during the current operating cycle through multimodal sensors, covering joint torque values, end effector acceleration values, and joint angle values, comprehensively reflecting the robotic arm's operating status. This data is preprocessed to generate a standardized dataset, providing high-quality input for the deep learning model. The trained deep learning model predicts the dynamic characteristics of the load in the next operating cycle, accurately grasping the load mass distribution and center of gravity position change trends. Based on the prediction results, the gravity compensation torque value is calculated using the robotic arm's dynamic model, making the compensation more accurate. The gravity compensation torque value is fed back to the control system in real time, adjusting the output torque of the drive motor to achieve dynamic gravity compensation control of the robotic arm. This effectively improves the motion accuracy of the robotic arm during operation, reduces positioning deviations and operational errors caused by gravity, improves production efficiency and product quality, and ensures stable and efficient operation of the robotic arm in complex load environments. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0014] Figure 1 This is a schematic diagram of a scenario for the robotic arm gravity dynamic compensation control method based on multimodal prediction in the embodiments of this application;

[0015] Figure 2 A flowchart of a method for dynamic gravity compensation control of a robotic arm based on multimodal prediction is provided for embodiments of this application;

[0016] Figure 3 A schematic diagram of the load dynamic characteristic prediction process provided in the embodiments of this application;

[0017] Figure 4 A schematic diagram illustrating the process of generating local features provided in this application embodiment;

[0018] Figure 5 A schematic diagram illustrating the process of generating global features provided in this application embodiment;

[0019] Figure 6 This is another flowchart illustrating the load dynamic characteristic prediction provided in an embodiment of this application;

[0020] Figure 7a A schematic diagram illustrating the training process of the machine learning model provided in this application embodiment;

[0021] Figure 7b A schematic flowchart for training set expansion provided in the embodiments of this application.

[0022] Figure 8 A schematic diagram illustrating the process of generating gravity compensation torque values ​​provided in this application embodiment;

[0023] Figure 9 A schematic diagram of the structure of a multimodal prediction-based dynamic gravity compensation control system for a robotic arm provided in an embodiment of this application;

[0024] Figure 10 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0026] This application provides a method and system for dynamic gravity compensation control of a robotic arm based on multimodal prediction, which will be described in detail below.

[0027] Taking an automotive parts assembly workshop as an example, this workshop contains a large number of parts of different specifications and shapes that need to be assembled. The robotic arm undertakes the important task of grasping, transporting, and assembling these parts. The workshop environment is complex, and the weight and center of gravity of the parts vary greatly depending on the type. Moreover, during the assembly process, the robotic arm's movement posture is constantly changing, and the load also changes dynamically.

[0028] like Figure 1As shown, in this scenario, the robotic arm is equipped with torque sensors, acceleration sensors, and angle sensors. Torque sensors are installed at the joints of the robotic arm to record the torque exerted on the joints in real time during movement. For example, when grasping heavy engine parts, the torque exerted on the joints increases significantly, and the torque sensor can accurately detect this change. Acceleration sensors are located at the end effector of the robotic arm and measure the acceleration of the end effector during the handling of parts. When the robotic arm starts or stops rapidly, the acceleration sensor can accurately capture changes in acceleration. Angle sensors are distributed at each joint to record the angle values ​​of the joints and determine the posture of the robotic arm. For example, when inserting parts into specific positions, precise control of the joint angles is crucial, and angle sensors provide accurate angle information.

[0029] The workshop also includes a data processing center that receives load information data from various sensors. This center centrally processes this data to generate a standardized dataset corresponding to the robotic arm's operating status. The standardized dataset is then input into a trained deep learning model, which predicts the robotic arm's load dynamics in the next operating cycle. Based on the load dynamics prediction, the center calculates the gravity compensation torque values ​​for each joint of the robotic arm in the next operating cycle and transmits these values ​​to the robotic arm's control system in real time. The control system adjusts the output torque of the drive motors for each joint based on the received gravity compensation torque values, thereby achieving dynamic gravity compensation control of the robotic arm in the next operating cycle. This ensures the robotic arm can accurately complete the assembly of parts and avoids assembly errors caused by gravity.

[0030] refer to Figure 2 , Figure 2 This is a flowchart illustrating a multimodal prediction-based dynamic gravity compensation control method for a robotic arm, as provided in an embodiment of this application. The execution entity of this method can be a computer device (which can serve as a data processing center). This computer device can be a single computer or a cluster of multiple computer devices. The computer device can be a terminal device or a server, etc. The multimodal prediction-based dynamic gravity compensation control method for a robotic arm provided in this embodiment specifically includes:

[0031] S10: Acquire load information data collected by the multimodal sensors during the current operating cycle of the robotic arm. The load information data includes joint torque values ​​recorded by the torque sensor, end-effector acceleration values ​​recorded by the acceleration sensor, and joint angle values ​​recorded by the angle sensor.

[0032] In this application, a multimodal sensor refers to a variety of sensors with different functional types that work together to collect information related to the load of the robotic arm from multiple dimensions.

[0033] A torque sensor is a device specifically designed to measure the torque applied to the joints of a robotic arm. Joint torque refers to the magnitude of the torsional force that a robotic arm joint needs to overcome to maintain a specific state of load, such as stationary, uniform motion, or variable-speed motion. For example, when a robotic arm grasps a heavy load, the joint needs to output a large torque to maintain the load's stability and achieve the desired movement; this torque value can be accurately measured by a torque sensor.

[0034] In this application, the joint torque value refers to the magnitude of the torsional force that a robotic arm joint needs to overcome to maintain a specific load state (such as stationary, uniform motion, or variable motion). It is a key parameter for measuring the force on the robotic arm joints. The joint torque value changes with the weight of the load, the position of the center of gravity, and the robotic arm's motion posture. For example, when a robotic arm grasps a heavy object, the joint needs to output a large torque to maintain the stability of the object and achieve the predetermined motion. This torque is accurately measured by a torque sensor. It reflects the intensity of the load's effect on the joint, helps to understand the force on each joint of the robotic arm, and is of great significance for analyzing the robotic arm's load and optimizing motion control strategies. Accurately obtaining the joint torque value can ensure that the robotic arm moves in the expected posture and avoid motion deviation and task failure due to improper torque.

[0035] Accelerometers are mainly used to detect the acceleration of the end effector of a robotic arm during operation.

[0036] In this application, the end-effector acceleration value is a physical quantity describing the rate of change of velocity at the end of the robotic arm, directly reflecting its dynamic characteristics during movement. When the robotic arm performs a task, its end-effector acceleration value changes accordingly during startup, shutdown, acceleration, and deceleration. For example, when the robotic arm quickly grasps a load, the end-effector rapidly reaches a certain speed from rest, at which point the acceleration value is relatively large; conversely, when approaching the target position and preparing to stop, the end-effector decelerates, and the magnitude and direction of the acceleration value change accordingly. This value not only reflects the changes in the robotic arm's own motion state but is also closely related to the inertia of the load. By monitoring and analyzing the end-effector acceleration value, we can gain a deeper understanding of the impact of the load on the robotic arm's motion, providing crucial information for precise control of the robotic arm's motion state. This helps optimize motion planning, avoid problems such as load swaying and inaccurate positioning caused by improper acceleration, and thus improve the stability and accuracy of the robotic arm's motion.

[0037] Angle sensors are used to record the angle values ​​of the robotic arm's joints, which determine the relative positions of the joints in space. Different combinations of joint angles determine the overall posture of the robotic arm. For example, in the process of accurately placing goods into a specific position on a shelf, precise control of the joint angles is crucial. Angle sensors can provide accurate angle information to ensure that the robotic arm moves along a predetermined trajectory.

[0038] In one embodiment, the torque sensor can be a strain gauge torque sensor, which works based on the fact that a strain gauge deforms under force, causing a change in resistance. The torque is calculated by measuring the change in resistance. The accelerometer can be a MEMS accelerometer, fabricated using silicon micromachining technology. It measures acceleration by detecting the inertial force generated by a mass under acceleration. The angle sensor can be a photoelectric encoder, which converts the rotation angle of the robotic arm joint into a digital signal output through photoelectric conversion. These sensors collect data in real time at a set frequency and transmit the load information data to the data processing unit via a high-speed data transmission line.

[0039] S20: Preprocess the load information data to generate a standardized dataset corresponding to the operating state of the robotic arm. The standardized dataset includes torque change curves, acceleration change curves, and angle change curves over a time series.

[0040] In this application, preprocessing is the initial processing of load information data acquired from multimodal sensors to improve data quality and make it more suitable for subsequent analysis and processing. Since data acquired from different types of sensors may differ in dimensions, numerical ranges, and data formats, and may contain noise or outliers, preprocessing aims to eliminate these differences and interferences, ensuring data consistency and reliability. The standardized dataset is a dataset formed after preprocessing and organized according to specific standards, containing time-series torque variation curves, acceleration variation curves, and angle variation curves.

[0041] In this application, the torque variation curve presents the relationship between joint torque values ​​and time in a graphical form. With time on the horizontal axis and joint torque values ​​on the vertical axis, it visually demonstrates the dynamic changes in torque experienced by each joint of the robotic arm during operation. For example, during the robotic arm's grasping and handling of objects, as the action progresses—such as initial grasping, movement, and placement—the joints need to overcome different resistances, and the torque values ​​will change accordingly. These changes are clearly reflected on the torque variation curve. By analyzing this curve, one can gain insight into the load conditions of the robotic arm at different times and the force trends of the joints, helping technicians understand the robotic arm's working state, promptly detect abnormal torque fluctuations, provide strong data support for predicting dynamic load characteristics, and facilitate advance adjustments to control strategies to ensure stable and efficient operation of the robotic arm.

[0042] The acceleration variation curve depicts the evolution of the end-effector acceleration value over time, with time on the x-axis and the end-effector acceleration value on the y-axis. This curve reflects the drastic changes in the motion state of the end-effector. When the robot arm starts rapidly, stops suddenly, or changes direction abruptly, the acceleration value fluctuates significantly, appearing as large swings on the curve. For example, in material handling scenarios, the acceleration variation curve clearly shows the rapid changes in acceleration during the robot arm's rapid approach to and departure from the material placement point.

[0043] Angle variation curves represent the trajectory of robotic arm joint angle values ​​over time, with time on the horizontal axis and joint angle values ​​on the vertical axis. It reflects the posture adjustments of each joint during operation. Different task requirements will cause different angle changes in the robotic arm joints. For example, in assembly tasks, the robotic arm joints need to be precisely adjusted to align with the installation position of parts, and the angle variation curve will show corresponding precise fluctuations.

[0044] In one embodiment, the collected joint torque, end-effector acceleration, and joint angle values ​​are first cleaned by setting reasonable data thresholds and using statistical analysis methods to remove outlier data points that significantly deviate from the normal range. Then, a normalization method is used to map data of different dimensions to the [0, 1] interval. For example, for joint torque values, normalization is performed using the formula (X - Xmin) / (Xmax - Xmin), where X is the original data value, and Xmin and Xmax are the minimum and maximum values ​​in the data set, respectively. For time series data, a moving average filter is used for smoothing to remove noise interference. Specifically, a fixed-length time window is set, the average value of the data within the window is calculated, and this average value is used to replace the data point at the center of the window, thus obtaining smoothed time series data. Finally, the processed data is plotted in chronological order as torque variation curves, acceleration variation curves, and angle variation curves to generate a standardized dataset.

[0045] S30: Input the standardized dataset into the trained deep learning model, and predict the load dynamic characteristics of the robotic arm in the next operating cycle based on the deep learning model. The load dynamic characteristics include load mass distribution characteristics and load center of gravity position change characteristics.

[0046] In this application, the deep learning model is a model based on an artificial neural network architecture with powerful data processing and pattern recognition capabilities. Through learning from a large amount of historical data, it can automatically extract complex features and patterns from the data, thereby establishing a mapping relationship between input data and output results.

[0047] In this embodiment, a standardized dataset is used as input, and the deep learning model aims to predict the load dynamics of the robotic arm in the next operating cycle. Load mass distribution characteristics describe the distribution of load mass on the robotic arm, such as whether the load is concentrated at one end or evenly distributed within the working range. This is crucial for analyzing the weight distribution of the load borne by each joint of the robotic arm. Load center of gravity position change characteristics reflect how the position of the load center of gravity changes in space over time as the robotic arm moves. This is critical for the balance control and motion planning of the robotic arm. Predicting these characteristics through a deep learning model allows for advance understanding of the dynamic trends of load changes, providing forward-looking guidance for the control of the robotic arm, enabling it to better adapt to load changes and improving the accuracy and stability of motion control.

[0048] In one embodiment, the deep learning model can employ a combination of the Transformer architecture and a Convolutional Neural Network (CNN). First, the CNN layer extracts local features from the time-series curves in the standardized dataset, capturing local patterns and features through sliding convolution operations on the data. Then, the extracted local features are input into the Transformer architecture, utilizing its self-attention mechanism to effectively handle long-range dependencies in the time-series data and generate a global feature representation. Finally, a fully connected layer maps the global feature representation to the prediction results of load mass distribution characteristics and load center-of-gravity position change characteristics. During training, a large amount of historical robotic arm operation data is used, and the model parameters are continuously adjusted through backpropagation to gradually reduce the prediction error until the preset accuracy requirements are met.

[0049] S40: Based on the prediction results of the load dynamic characteristics, calculate the gravity compensation torque value of each joint of the robotic arm in the next operating cycle. The gravity compensation torque value is obtained by combining the robotic arm dynamic model with the load dynamic characteristics.

[0050] In this application, the robotic arm dynamics model is a mathematical model based on mechanical principles, comprehensively considering factors such as the robotic arm's structure, mass distribution, joint motion, and external forces, describing the relationship between the robotic arm's motion and forces. The predicted results of the load dynamic characteristics, namely the load mass distribution characteristics and the load center of gravity position change characteristics, provide crucial basis for calculating the gravity compensation torque value. The gravity compensation torque value refers to the additional torque required to be applied to each joint to counteract the influence of gravity on the robotic arm's motion, enabling the robotic arm to move accurately according to the expected trajectory and posture. For example, when the load center of gravity position changes, the distribution and magnitude of the gravity force on each joint of the robotic arm will change. By calculating the gravity compensation torque value, the driving torque of each joint can be adjusted to maintain the balance and stable motion of the robotic arm. Accurately calculating the gravity compensation torque value is crucial for improving the motion accuracy and control performance of the robotic arm, ensuring that the robotic arm can accurately complete tasks under different loads and motion postures, and avoiding positional deviations and motion instability caused by gravity.

[0051] In one embodiment, a Lagrange dynamics method is used to establish a dynamic model of the robotic arm. The robotic arm is considered as a system composed of multiple rigid bodies connected by joints. Based on the Lagrange equations, considering the mass distribution of the load and the position of the center of gravity, the generalized forces of each joint of the robotic arm are calculated. When calculating the gravity compensation torque, the predicted dynamic characteristics of the load are substituted into the dynamic model. By solving the corresponding dynamic equations, the gravity compensation torque value of each joint in the next operating cycle is obtained. Specifically, the equivalent mass borne by each joint is determined based on the load mass distribution characteristics. The lever arm of gravity acting on each joint is calculated by combining the load center of gravity position change characteristics. Then, the gravity compensation torque value of each joint is obtained through the torque calculation formula (torque = force × lever arm). This calculation method based on the dynamic model can fully consider the actual structure and load characteristics of the robotic arm, improving the accuracy of the gravity compensation torque value calculation.

[0052] First, a dynamic model of the robotic arm is established. Using Lagrange dynamics, the robotic arm is considered as a system composed of multiple rigid bodies connected by joints. For a robotic arm with n joints, the mass, inertia tensor, and joint variables of each rigid body are defined. Based on the Lagrange equations... (in As the system's kinetic energy, (where the system potential energy is used), the dynamic equation of the robotic arm is derived.

[0053] The system's kinetic energy T is the sum of the kinetic energies of each rigid body. The kinetic energy of each rigid body is related to its velocity and angular velocity, which in turn are related to the joint variables and their derivatives. The system's potential energy V mainly consists of gravitational potential energy, which is related to the positions of each rigid body and the gravity of the load. By taking the partial derivatives of L with respect to the joint variables and their derivatives, and combining this with the definition of generalized force, the dynamic equations of the robotic arm are obtained:

[0054]

[0055] in, It is the joint torque vector. It is a joint variable vector. and These are the joint velocity vector and the joint acceleration vector, respectively. It is the inertia matrix. It is the matrix of Coriolis force and centrifugal force. It is the gravity vector.

[0056] Next, we consider the dynamic characteristics of the load. These characteristics include the load mass distribution characteristics and the load center of gravity position change characteristics. Based on the load mass distribution characteristics, we determine the equivalent load mass borne by each joint. For example, by analyzing the distance between the load and each joint and the force transmission relationship, we proportionally distribute the load mass to the rigid bodies corresponding to each joint, thereby affecting the inertia matrix. Make corrections.

[0057] The characteristics of load center of gravity position change are converted into additional torques acting on each joint. As the load center of gravity position changes, the direction and magnitude of the gravity acting on the joints change, resulting in dynamic gravitational effects. Using kinematic relationships, the velocity and acceleration of the load center of gravity at different times are calculated. Combined with the load mass, the additional torque generated at each joint due to the change in the load center of gravity position is calculated using the Newton-Euler equations.

[0058] Finally, the gravity compensation torque values ​​are calculated. In each operating cycle, the parameters in the robotic arm's dynamics equations are updated based on the current load dynamics. Then, based on the robotic arm's desired motion state (e.g., stationary, uniform motion, or motion along a specific trajectory), the gravity compensation torque values ​​required to counteract the effects of gravity are calculated for each joint. For example, when the robotic arm is expected to remain stationary, the joint acceleration... and joint velocity When the torque is zero, the gravity compensation torque is mainly used to balance the gravity vector. And the additional torque generated due to the dynamic characteristics of the load. In this way, the gravity compensation torque value of each joint of the robotic arm in the next operating cycle can be accurately calculated, providing a precise basis for realizing the gravity dynamic compensation control of the robotic arm.

[0059] S50: The gravity compensation torque value is fed back to the control system of the robotic arm in real time, and the output torque of the drive motor of each joint of the robotic arm is adjusted to realize the dynamic gravity compensation control of the robotic arm in the next operating cycle.

[0060] In this application, the control system of the robotic arm is responsible for receiving various control signals and precisely controlling the movement of the robotic arm based on these signals. The gravity compensation torque value is fed back to the control system in real time, allowing the control system to adjust the output torque of the drive motors of each joint of the robotic arm based on this information. The drive motor is the device that provides power to the joints of the robotic arm; by changing the output torque, the joints of the robotic arm rotate accordingly, thereby realizing the movement of the robotic arm. Achieving dynamic gravity compensation control of the robotic arm in the next operating cycle can be achieved by adjusting the output torque of the drive motor, enabling the robotic arm to counteract the influence of gravity in real time during operation, maintaining a stable and accurate movement state. For example, when a change in load causes a change in the effect of gravity on the robotic arm, timely adjustment of the drive motor output torque can ensure that the robotic arm can still move along the predetermined trajectory, improving the working efficiency and movement accuracy of the robotic arm and meeting the high-precision requirements in actual production.

[0061] In one embodiment, reference Figure 3 Step S30 can specifically include the following process:

[0062] S301: Extract time series features from the standardized dataset, the time series features including torque rate of change, acceleration rate of change and angle rate of change.

[0063] In this application, time series features are feature parameters mined from standardized datasets that reflect the trend and regularity of data changes over time. The torque change rate represents the degree of change in joint torque values ​​per unit time, reflecting how quickly the joint torque changes during the operation of the robotic arm. For example, at the instant the robotic arm grasps a load, the joint torque may increase rapidly; the torque change rate can quantify the degree of this rapid change, helping to analyze the dynamic process of the load's influence on the joint torque. The acceleration change rate is the change in the robotic arm's end-effector acceleration per unit time, demonstrating the drastic degree of change in the robotic arm's end-effector motion state. For example, when the robotic arm starts or stops rapidly, the acceleration change rate can reflect the drastic degree of this speed change, helping to understand the dynamic influence of changes in the robotic arm's motion state and the load's inertia on acceleration. The angle change rate describes the change in the robotic arm's joint angles per unit time, used to analyze changes in the robotic arm's joint rotation speed, and is helpful in determining the rate and process of the robotic arm's posture adjustment.

[0064] In one embodiment, for the torque variation curve, the torque change rate can be obtained by calculating the ratio of the difference in torque values ​​at adjacent time points to the time interval. To reduce the impact of noise on the calculation results, the torque variation curve can be smoothed first, for example, using the Savitzky-Golay filtering method, which achieves smoothing by performing polynomial fitting on the data within a local window. The same calculation method is used for the acceleration variation curve and the angle variation curve to obtain the acceleration change rate and the angle change rate, respectively. Meanwhile, to ensure the accuracy and stability of the calculation, the size of the time interval can be reasonably adjusted according to the characteristics of the data and actual needs.

[0065] S302: Input the time series features into the first convolutional layer of the deep learning model to extract local feature vectors.

[0066] In this application, the first convolutional layer of the deep learning model is specifically designed to extract local features from the input data. The convolutional layer performs sliding convolution operations on the input data using convolutional kernels, automatically capturing local patterns and features within the data. The local feature vector is the result of the convolutional layer processing the time-series features, containing key information about the time-series data within a local scope. For time-series features composed of torque change rate, acceleration change rate, and angle change rate, the convolutional kernel performs convolution operations on the feature data within each local time window during the sliding process, thereby extracting the combined features of these features in local time and dimension.

[0067] In one embodiment, the kernel size of the first convolutional layer can be selected based on the characteristics of the data and experimental results, for example, a 3×1 kernel. The time-series features are arranged in a certain order into a two-dimensional matrix as input to the convolutional layer. The convolution kernel slides row by row on the input matrix, performing convolution operations on the data within each local time window. The results of the convolution operations are accumulated to obtain the local feature map. The convolution operation here can be understood as multiplying the convolution kernel by the corresponding elements of the data within the local time window and then summing the results. Then, non-linear activation processing is applied to the local feature map, for example, using the ReLU (Rectified Linear Unit) function, whose expression is f(x) = max(0, x). The ReLU function enhances the non-linear expressive power of the features, enabling the model to learn more complex feature relationships and generate preliminary local feature vectors. Finally, the preliminary local feature vectors are input to the pooling layer, where max pooling extracts the most representative local feature vectors for use in the subsequent global feature generation of the Long Short-Term Memory (LSTM) network layer. Max pooling selects the maximum value within a local region as the output, which can reduce data dimensionality and computational cost while preserving key features.

[0068] In one embodiment, reference Figure 4 Step S302 may specifically include the following steps:

[0069] S3021: Obtain time series features from the standardized dataset, including torque change rate, acceleration change rate, and angle change rate.

[0070] In one embodiment, torque, acceleration, and angle data are read sequentially from a standardized dataset in chronological order. For torque data, the rate of change of torque is obtained by calculating the ratio of the difference in torque values ​​at adjacent time points to the time interval. For example, assuming that the torque values ​​are M1 and M2 at times t1 and t2 (t2 > t1), respectively, and the time interval is Δt = t2 - t1, then the rate of change of torque is (M2 - M1) / Δt. To improve the accuracy of the calculation, the torque data can be smoothed, such as by using a moving average method to calculate the average value within a shorter time window before calculating the rate of change. For acceleration and angle data, the same method is used to calculate the rate of change of acceleration and the rate of change of angle, respectively. In this way, time series features are accurately obtained from the standardized dataset, laying the foundation for subsequent feature extraction and analysis.

[0071] S3022: Divide the time series features into multiple local time windows, each local time window containing time series data of a fixed length.

[0072] In this application, each local time window contains time-series data of a fixed length. This division helps to capture the characteristics and patterns of the data within a local time period. For example, when analyzing the operating state of a robotic arm, different local time windows may correspond to different motion stages of the robotic arm, such as the start-up stage, the stable motion stage, or the stopping stage. By processing the data within each local time window separately, features related to specific motion stages can be extracted more accurately, providing more targeted information for subsequent model training and prediction. Moreover, the fixed-length time window setting ensures data consistency and comparability, which is beneficial for model learning and analyzing patterns in the data.

[0073] In one embodiment, a suitable local time window length is determined based on the characteristics of the robotic arm's motion and the data sampling frequency. For example, if the data sampling frequency is 100Hz, i.e., 100 data points are collected per second, the local time window length can be set to 50 data points, corresponding to a time length of 0.5 seconds. Following this length, fixed-length data segments are sequentially extracted as local time windows, starting from the beginning of the time series feature. For example, for a time series of torque change rate, starting from the first data point, data points 1 to 50 are extracted as the first local time window, data points 51 to 100 as the second local time window, and so on. This divides the entire time series feature into multiple local time windows of fixed length, providing a suitable data structure for subsequent convolution operations.

[0074] S3023: For each local time window, a sliding convolution operation is performed on the time series data through the convolution kernel of the first convolutional layer to generate a local feature map.

[0075] In one embodiment, the convolution kernel size is assumed to be 3×1 with a stride of 1. For time-series data within a local time window (such as three-dimensional data composed of torque rate of change, acceleration rate of change, and angle rate of change), the convolution kernel starts from the top left corner of the data and performs multiplication operations with the corresponding three data points. The results are then accumulated to obtain a value in the local feature map. Next, the convolution kernel moves one data point to the right according to the stride, repeating the above operation until the entire width of the local time window has been traversed. Then, the convolution kernel moves down one row (in the case of three-dimensional data) and continues the sliding convolution operation, ultimately generating a local feature map associated with the local time window. In this way, convolution operations are performed on the time-series data within each local time window to extract local features and generate the corresponding local feature map.

[0076] S3024: Perform nonlinear activation processing on the local feature map to generate a preliminary local feature vector.

[0077] In one embodiment, the ReLU function is used to perform nonlinear activation processing on the local feature map. For each element x in the local feature map, after processing by the ReLU function, the output value is y = max(0, x). Thus, processing all elements in the local feature map with the ReLU function yields a new vector, i.e., the initial local feature vector. This nonlinear activation processing not only enhances the model's ability to represent nonlinear features in the data but also alleviates the gradient vanishing problem to some extent, which is beneficial for model training and convergence. It lays the foundation for subsequently generating more representative local feature vectors and accurately predicting load dynamics.

[0078] S3025: Input the preliminary local feature vector into the pooling layer, and extract the most representative local feature vector through max pooling operation for use in the subsequent global feature generation of the long short-term memory network layer.

[0079] In this application, max pooling is a method in pooling layers that selects the maximum value within a local region as the output. By inputting the initial local feature vector into the pooling layer and performing max pooling, the most representative features can be extracted from the initial local feature vector, reducing data dimensionality, lowering computational cost, and highlighting important feature information in the data.

[0080] In one embodiment, it is assumed that the initial local feature vector is divided into multiple non-overlapping local regions, each region being 2×2 in size (which can be adjusted according to actual conditions). Within each local region, the values ​​of four elements are compared, and the maximum value is selected as the output of that region. For example, for the elements [a, b, c, d] in a certain 2×2 local region, after the max pooling operation, the output value is max(a, b, c, d). By performing max pooling on all such local regions in the initial local feature vector, a vector with reduced dimensionality but retaining key features is obtained, namely the most representative local feature vector.

[0081] S303: Input the local feature vector into the long short-term memory network layer of the deep learning model to generate a global feature vector.

[0082] In this application, the Long Short-Term Memory (LSTM) network layer is a special network structure in deep learning models that excels at handling long-term dependencies in time series data. While local feature vectors contain key features of the time series data at a local level, the LSTM layer needs to integrate these local features to comprehensively understand the information contained in the entire time series. The global feature vector, generated by the LSTM layer after processing the local feature vectors, integrates information from different time steps of the time series and reflects the overall characteristics and long-term trends of the data.

[0083] In one embodiment, reference Figure 5 Step S303 can be implemented in the following way:

[0084] S3031: Arrange the local feature vectors in chronological order to form a local feature sequence.

[0085] In this application, by arranging local feature vectors in chronological order, the position of each local feature vector in the sequence corresponds to its order on the time axis of the robotic arm's operation. In this way, the LSTM layer can process local features in chronological order, understand the evolution of the robotic arm's operating state over time, and thus generate global feature vectors more accurately, ultimately improving the prediction accuracy of load dynamic characteristics.

[0086] In one embodiment, it is assumed that a series of local feature vectors have been obtained, for example, local feature vectors v1, v2, v3...vn are generated sequentially in time during a complete operation of the robotic arm. These vectors are arranged in order into a sequence [v1, v2, v3, ..., vn], where v1 corresponds to local features in the early stages of the robotic arm's operation, and vn corresponds to local features in later stages. This temporal arrangement forms a sequence of local features, which is provided as input to the Long Short-Term Memory (LSTM) network layer so that the LSTM layer can perform subsequent feature processing and analysis based on the time dimension.

[0087] S3032: Input the local feature sequence into the Long Short-Term Memory network layer, and capture the long-term dependencies in the time series through the memory units of the network layer.

[0088] In this application, the memory unit in the Long Short-Term Memory (LSTM) network layer is the core component that enables it to effectively handle long-term dependencies in time-series data. When local feature sequences are input into the LSTM layer, the memory unit selectively remembers and forgets the input local features through its unique gating mechanism, thereby capturing long-term dependencies in the time series. These long-term dependencies reflect the correlation of the robotic arm's operating state over a long time span, such as how changes in the robotic arm's load at different stages affect subsequent operating states.

[0089] In one embodiment, the memory unit includes an input gate, a forget gate, and an output gate. When a local feature vector from the local feature sequence is input, the input gate determines how much information from the currently input local feature can enter the memory unit; the forget gate determines how much previously stored information in the memory unit needs to be retained; and the output gate determines the information to be output to the next layer based on the current state of the memory unit. For example, when the load state of the robotic arm changes slowly, the forget gate may retain more previously remembered information, while the input gate allows new local feature information to enter to update the state of the memory unit, thereby accurately capturing long-term dependencies in the time series.

[0090] S3033: At each time step, update the state value of the memory cell, which includes the activation values ​​of the input gate, forget gate, and output gate.

[0091] In this application, during the processing of local feature sequences in the Long Short-Term Memory (LSTM) network layer, each time step corresponds to processing a local feature vector within the local feature sequence. At each time step, the memory unit updates its own state value based on the currently input local feature vector and the state value of the previous time step. The state value includes the activation values ​​of the input gate, forget gate, and output gate. These gate activation values ​​determine how the memory unit processes information: the input gate activation value controls the degree to which current input information enters the memory unit; the forget gate activation value determines the proportion of information retained from the previous time step; and the output gate activation value determines the information content output by the memory unit to the next layer. By continuously updating these state values, the memory unit can dynamically adapt to the input local feature sequence, accurately capture long-term dependencies in the time series, and thus continuously adjust its own state to reflect the changes in the robotic arm's operating state over time.

[0092] In one embodiment, assume the local feature vector of the input at the current time step is xt, the state value of the memory unit at the previous time step is Ct - 1, and the hidden state is ht - 1. The activation value it of the input gate is calculated through a linear transformation involving a weight matrix and a sigmoid function: Where Wix and Wih are weight matrices, and bi is the bias term. The activation value ft of the forget gate is calculated in a similar manner: The activation value ot of the output gate is also calculated using a linear transformation and the sigmoid function: Based on these activation values, the memory unit updates its state value Ct, thereby dynamically processing the local feature vector of the input at each time step and capturing long-term dependencies in the time series.

[0093] S3034: Generate the hidden state vector of the current time step based on the state value of the memory unit.

[0094] In this application, the hidden state vector is a vector generated by the Long Short-Term Memory (LSTM) network layer at each time step based on the state values ​​of the memory units. It integrates the local features of the current input with the historical information stored in the memory units. The hidden state vector contains key feature representations of the robotic arm's operating state at the current time step. These features reflect both the information carried by the current local feature vector and the historical information captured by the memory units through long-term dependencies. By generating the hidden state vector, the LSTM layer can effectively pass the state information of the memory units to the next layer, providing an important intermediate representation for the subsequent generation of global feature vectors. This helps to comprehensively and accurately describe the characteristics of the robotic arm's operating state over time, thereby improving the predictive ability of load dynamics.

[0095] In one embodiment, the hidden state vector ht for the current time step is generated based on the state value Ct of the memory cell and the activation value ot of the output gate. First, a nonlinear transformation is performed on the state value Ct of the memory cell, for example, using the tanh function: Ct' = tanh(Ct). Then, the hidden state vector ht is obtained by multiplying the output gate activation value ot with the transformed memory cell state value Ct'. The resulting hidden state vector ht contains important information related to the current time step from the memory unit, and through the control of the output gate, it selectively outputs features valuable for subsequent processing, providing a crucial intermediate step for generating the global feature vector.

[0096] S3035: Concatenates the hidden state vectors of all time steps to form a global feature vector, which is used to predict the load dynamic characteristics of the fully connected layer output of the deep learning model.

[0097] In one embodiment, it is assumed that during the processing of the local feature sequence, n hidden state vectors h1, h2, ..., hn are obtained. These hidden state vectors are concatenated sequentially to form a higher-dimensional vector H = [h1, h2, ..., hn], which is the global feature vector. The global feature vector H is then input into the fully connected layer of the deep learning model. The fully connected layer processes it through linear transformations and nonlinear activation functions to further extract high-level features, ultimately outputting the predicted results of load dynamic characteristics, such as load mass distribution characteristics and load center of gravity position change characteristics. This helps the robotic arm better cope with dynamic load changes and achieve precise gravity compensation control.

[0098] S304: Based on the global feature vector, the fully connected layer of the deep learning model outputs the load dynamic characteristics of the robotic arm in the next operating cycle. The load dynamic characteristics include load mass distribution characteristics and load center of gravity position change characteristics.

[0099] In this application, the fully connected layer is the part of the deep learning model used to integrate the features extracted by the previous layers and output the final prediction result. The global feature vector contains long-term dependency information in the time series. The fully connected layer processes the global feature vector through linear transformation and non-linear activation functions to further extract high-level features, thereby generating the prediction result of the load dynamic characteristics.

[0100] In one embodiment, reference Figure 6 Step S304 can be implemented in the following way:

[0101] S3041: Obtain the global feature vector, which contains long-term dependency information in the time series.

[0102] In this application, the long-term dependency information in the global feature vector reflects the correlation between the operating states of the robotic arm at different points in time. For example, how early changes in the load state affect subsequent movements, and the interaction of factors such as joint angles, torques, and accelerations over long periods. By obtaining this global feature vector, the deep learning model can gain a comprehensive understanding of the dynamic changes during the operation of the robotic arm.

[0103] S3042: Input the global feature vector into the first hidden layer of the fully connected layer, and generate intermediate feature representations through linear transformation and nonlinear activation functions.

[0104] In this application, the first hidden layer of the fully connected layer further processes the input global feature vector, generating intermediate feature representations through linear transformation and nonlinear activation functions. The linear transformation maps the global feature vector to a new feature space by multiplying the weight matrix with the global feature vector and adding a bias term, thus changing the dimension and representation of the features. The nonlinear activation function introduces nonlinear factors, enhancing the model's expressive power and enabling it to learn more complex feature relationships.

[0105] In one embodiment, assume the global feature vector is x, the weight matrix of the first hidden layer is W1, and the bias term is b1. First, a linear transformation is performed to obtain... Then, the result of the linear transformation, y, is input into a non-linear activation function, such as the ReLU function, to obtain the intermediate feature representation z = ReLU(y). Through this operation, the global feature vector undergoes linear transformation and non-linear activation processing in the first hidden layer, generating a more representative intermediate feature representation.

[0106] S3043: Input the intermediate feature representation into the second hidden layer of the fully connected layer to further extract high-level features.

[0107] In this application, the intermediate feature representation, after processing by the first hidden layer, already contains a certain degree of abstract features. However, in order to more accurately predict load dynamics, it is necessary to further mine and refine features in the second hidden layer. By performing linear transformation and nonlinear activation operations again, the second hidden layer can capture more complex and higher-level feature relationships. These high-level features are crucial for accurately describing the load mass distribution characteristics and the load center of gravity position change characteristics, which helps to improve the model's prediction accuracy of load dynamics.

[0108] In one embodiment, let the intermediate feature be represented as z, the weight matrix of the second hidden layer be W2, and the bias term be b2. First, a linear transformation is performed to obtain... Next, 'a' is input into a non-linear activation function, such as the Sigmoid function, to obtain a high-level feature representation s = Sigmoid(a). Through this series of operations, the second hidden layer extracts higher-level features from the intermediate feature representation. These features can more accurately reflect the relationship between load dynamics and the robotic arm's operating state, providing crucial support for ultimately generating accurate load dynamics prediction results.

[0109] S3044: Based on the high-level features, load dynamic characteristics prediction results are generated through the output layer of the fully connected layer. The load dynamic characteristics include load mass distribution characteristics and load center of gravity position change characteristics.

[0110] In this application, the output layer of the fully connected layer generates load dynamic characteristic prediction results based on the high-level features extracted by the second hidden layer. The output layer maps the high-level features to the dimensional space of load dynamic characteristics through linear transformation, thereby obtaining predicted values ​​of load mass distribution characteristics and load center of gravity position change characteristics. These prediction results are learned by the deep learning model based on the time series data of the robotic arm's operating state, reflecting the prediction of future dynamic changes in the load.

[0111] In one embodiment, let the high-level feature representation be s, the weight matrix of the output layer be W3, and the bias term be b3. The prediction result is obtained through linear transformation. Here, p contains the predicted values ​​of the load mass distribution characteristics and the load center of gravity position change characteristics. For example, the prediction result p can be a vector, where some elements represent relevant parameters of the load mass distribution, and other elements represent the predicted coordinates or parameters of the load center of gravity position change. In this way, the output layer of the fully connected layer generates the load dynamic characteristic prediction results based on high-level features, providing crucial decision information for the actual operation control of the robotic arm.

[0112] refer to Figure 7a The training process of the deep learning model in this application embodiment includes the following steps:

[0113] S60: Acquire load information data of the robotic arm in multiple historical operating cycles. The load information data includes joint torque values ​​recorded by the torque sensor, end-effector acceleration values ​​recorded by the acceleration sensor, and joint angle values ​​recorded by the angle sensor.

[0114] In this application, the load information data within multiple historical operating cycles refers to the set of data reflecting the load status of the robotic arm collected by torque sensors, acceleration sensors, and angle sensors during a series of past operating cycles.

[0115] In one embodiment, in order to acquire this data, an appropriate sampling frequency can be set during the long-term operation of the robotic arm, such as 50 times per second, to ensure that subtle changes in the robotic arm's operating state can be captured.

[0116] S61: Label the load information data to generate a training dataset. The labeled information in the training dataset includes load quality distribution characteristics and load center of gravity position change characteristics.

[0117] In one embodiment, detailed physical parameters of the load carried by the robotic arm in each historical operating cycle are first obtained. These parameters can be obtained by consulting the product specifications of the load, measuring the load mass using a weighing device, and obtaining the coordinates of the load's center of gravity using 3D laser scanning technology. Based on these load physical parameters, the theoretical load mass distribution characteristics and theoretical load center of gravity position change characteristics of the robotic arm in each historical operating cycle are calculated using mechanical analysis software or theoretical calculation formulas. For example, based on the shape and mass of the load and the motion posture of the robotic arm, the change in the load's center of gravity position is determined using a center of gravity calculation formula. Then, these theoretical load dynamic characteristics are used as annotation information and associated with the corresponding load information data to form a preliminary training dataset.

[0118] S62: Divide the training dataset into a training subset and a validation subset, use the training subset to train the initial deep learning model, and use the validation subset to evaluate the performance of the trained deep learning model.

[0119] In this application, the training subset is used to train the initial deep learning model. During the training process, the model continuously adjusts its parameters and learns the relationship between the load information and the labeled information in the training dataset, thereby gradually optimizing the model's performance. The validation subset is used to evaluate the model's performance after training. It can test the model's predictive ability on unseen data and help determine whether the model is overfitting or underfitting.

[0120] In one embodiment, a stratified sampling method is used to divide the training dataset into a training subset and a validation subset in a 7:3 ratio. This ensures that the training and validation subsets have similar data distributions, containing data samples under various load conditions and robotic arm operating states. When training the initial deep learning model using the training subset, a stochastic gradient descent algorithm is used to update the model parameters. Each time, a batch of data is randomly selected from the training subset as a training batch, and the model adjusts its parameters based on the error between the prediction results of this batch of data and the labeled information. During training, the model is periodically evaluated using the validation subset, and the prediction error of the model on the validation subset, such as the mean squared error (MSE), is calculated using the following formula: Where yi is the actual value, Here, n is the predicted value, and n is the number of samples. The model is considered to have achieved good training results when the prediction error on the validation subset no longer decreases significantly.

[0121] S63: Based on the performance evaluation results, adjust the hyperparameters of the deep learning model until the prediction error of the deep learning model meets the preset accuracy requirements.

[0122] In this application, the hyperparameters of the deep learning model are parameters pre-set before model training, which determine the model's structure and training method. Performance evaluation results reflect the model's current predictive ability, and adjusting the hyperparameters based on this can optimize model performance. For example, hyperparameters such as the number of layers in the neural network, the number of neurons, and the learning rate all affect the model's complexity and learning speed.

[0123] In one embodiment, a hyperparameter search space can be established, including different learning rates (e.g., 0.001, 0.01, 0.1), the number of neural network layers (e.g., 2, 3, 4 layers), and the number of neurons (e.g., 10, 20, 30). A random search algorithm is used to randomly select hyperparameter combinations from this search space for model training and validation. After each training iteration, the prediction error of the model on the validation subset is recorded. Based on the results of multiple trials, the hyperparameter combination with the smallest prediction error and meeting the preset accuracy requirements is selected as the final model hyperparameter settings.

[0124] In one embodiment, reference Figure 7b In this embodiment of the application, step S610 can be implemented in the following way:

[0125] S610: Obtain the load physical parameters of the robotic arm in each historical operating cycle, including the load mass value and the coordinates of the load center of gravity.

[0126] In this application, the physical parameters of the load are crucial parameters describing the inherent properties and spatial position characteristics of the load. The load mass directly determines the weight carried by the robotic arm; different load masses cause the joints of the robotic arm to experience varying degrees of force. For example, when the robotic arm handles large equipment, the load mass is large, and the joints need to withstand greater torque to maintain balance and achieve movement; while when handling small parts, the load mass is small, and the torque on the joints is relatively small. The coordinates of the load's center of gravity determine the position of the equivalent point of application of the load's gravity in space, which changes with the movement of the robotic arm and the change of the load's own posture. For example, when the robotic arm grasps an irregularly shaped object and changes its posture, the position of the load's center of gravity will change accordingly.

[0127] In one embodiment, for loads with regular shapes and uniform mass distribution, the load mass value can be obtained by consulting the product specifications, and the coordinates of the load's center of gravity can be calculated based on the geometry. For loads with irregular shapes or uneven mass distribution, the load mass value is directly measured using a high-precision weighing device. To obtain the coordinates of the load's center of gravity, a suspension method can be used. The load is suspended by a thin thread, and under the influence of gravity, the intersection point of the extensions of the suspension thread is marked to determine the position of the center of gravity in a two-dimensional plane. By changing the suspension point multiple times, the coordinates of the center of gravity in three-dimensional space can be finally determined. Alternatively, advanced three-dimensional laser scanning technology can be used to acquire three-dimensional point cloud data of the load, and algorithms can be used to process this data to accurately calculate the coordinates of the load's center of gravity.

[0128] S611: Based on the load physical parameters, calculate the theoretical load dynamic characteristics of the robotic arm in each historical operating cycle. The theoretical load dynamic characteristics include the theoretical load mass distribution characteristics and the theoretical load center of gravity position change characteristics.

[0129] In this application, the theoretical load dynamic characteristics are derived from the physical parameters of the load through theoretical analysis and calculation, reflecting the dynamic changes of the load during the operation of the robotic arm. The theoretical load mass distribution characteristics describe the theoretical distribution of the load mass on the robotic arm. For example, if the load mass is concentrated at one end of the robotic arm, the joints closer to that end will bear greater load pressure. The theoretical load center of gravity position change characteristics reflect the theoretical position change of the load center of gravity in space as the robotic arm moves, playing a crucial role in the balance control and motion planning of the robotic arm. By calculating the theoretical load dynamic characteristics, accurate annotation information can be provided for the training dataset, enabling the deep learning model to learn the inherent laws of load dynamic changes, thereby improving the accuracy and reliability of the model's prediction of load dynamic characteristics.

[0130] S612: The theoretical load dynamic characteristics are used as annotation information and associated with the corresponding load information data to generate a preliminary training dataset.

[0131] In one embodiment, using time series data as an index, the load information data (joint torque values, end-effector acceleration values, joint angle values) for each historical operating cycle are mapped one-to-one with the corresponding theoretical load mass distribution characteristics and theoretical load center of gravity position change characteristics. A database management system can be used to store this data, creating a data table where each row represents a data record for one historical operating cycle, and each column corresponds to different load information data and annotation information. For example, the first column stores joint torque values, the second column stores end-effector acceleration values, the third column stores joint angle values, and the fourth and fifth columns store relevant parameters for the theoretical load mass distribution characteristics and theoretical load center of gravity position change characteristics, respectively. This data storage structure facilitates data retrieval and processing during subsequent model training.

[0132] S613: Perform data augmentation processing on the initial training dataset to generate an expanded training dataset. The data augmentation processing includes data smoothing and noise injection.

[0133] In this application, data augmentation processing aims to improve the generalization ability of deep learning models by increasing the diversity and richness of the data through a series of operations on the initial training dataset.

[0134] In one embodiment, a Gaussian smoothing filter is used to process the data for smoothing. An appropriate Gaussian kernel size is selected based on the characteristics of the data and the noise level. For example, for time-series data of joint torque values, assuming that the noise is mainly concentrated in the high-frequency part, a Gaussian kernel of size 5 is selected, with its weight distribution conforming to a Gaussian function. The Gaussian kernel is convolved with the time-series of joint torque values ​​to smooth each data point, making the changes between adjacent data points smoother and effectively removing noise. A similar method is used for smoothing acceleration and angle data.

[0135] S614: Divide the extended training dataset into a training subset and a validation subset for training and evaluation of the deep learning model.

[0136] In one embodiment, a stratified sampling method is used for partitioning. Based on the distribution of different load conditions and robotic arm operating states in the extended training dataset, the dataset is divided into a training subset and a validation subset in a certain ratio (e.g., 7:3). This ensures that the training and validation subsets have similar data distributions, meaning both subsets contain data samples with various load qualities, center of gravity positions, and robotic arm motion states. During model training, the model is iteratively trained using the training subset. After each iteration, the model performance is evaluated using the validation subset, observing changes in metrics such as prediction error and accuracy on the validation subset. Based on the evaluation results, the model's training strategy is adjusted, such as adjusting the learning rate and optimizing the network structure, until the model's performance on the validation subset reaches a satisfactory level, ensuring that the model can accurately predict load dynamics in practical applications.

[0137] In one embodiment, reference Figure 8 Step S50 can be implemented in the following way:

[0138] S501: Based on the load mass distribution characteristics in the load dynamic characteristics, calculate the static gravity compensation torque value of each joint of the robotic arm.

[0139] In this application, the load mass distribution characteristics reflect the distribution of the load mass on the robotic arm. The static gravity compensation torque value refers to the compensation torque that each joint needs to apply to counteract the torque generated by the joints due to the load gravity when the robotic arm is stationary or in uniform motion. For example, when the load mass is concentrated at one end of the robotic arm, the joints near that end need to withstand a greater static gravity, and correspondingly require a larger static gravity compensation torque to maintain balance. By accurately analyzing the load mass distribution characteristics, the static gravity compensation torque value required for each joint can be calculated more precisely, thereby effectively counteracting the influence of gravity on the joints of the robotic arm and improving the stability and positioning accuracy of the robotic arm in static or uniform motion.

[0140] In one embodiment, based on the structural model of the robotic arm, the robotic arm is divided into multiple rigid body segments, each segment corresponding to a joint. Based on the load mass distribution characteristics, the equivalent load mass borne by each joint is determined. For example, for a robotic arm with six joints, by analyzing the distance between the load and each joint and the force transmission relationship, the load mass is proportionally distributed to each joint using static principles. Then, based on the distance from each joint to the equivalent center of mass of the load and the gravitational acceleration, the static gravity compensation torque value of each joint is calculated. The specific calculation formula is: Static gravity compensation torque value = Equivalent load mass borne by the joint × Gravitational acceleration × Distance from the joint to the equivalent center of mass of the load.

[0141] S502: Based on the load center of gravity position change characteristics in the load dynamic characteristics, calculate the dynamic gravity compensation torque values ​​of each joint of the robotic arm.

[0142] In this application, the load center of gravity position change characteristics describe the dynamic changes of the load center of gravity in space as the robotic arm moves. The dynamic gravity compensation torque value is the torque required to be applied to each joint to compensate for the additional forces on the joints caused by the change in the load center of gravity position. When the robotic arm moves, the change in the load center of gravity position causes changes in the direction and magnitude of the gravity force on the joints, resulting in dynamic gravity effects. For example, when the robotic arm starts, stops, or changes direction of movement quickly, the inertia of the load center of gravity will cause the joints to be subjected to additional forces. By calculating the dynamic gravity compensation torque value based on the load center of gravity position change characteristics, the robotic arm can effectively counteract this dynamic gravity effect during movement, ensuring the smoothness and accuracy of the robotic arm's movement and avoiding movement deviations caused by dynamic changes in gravity.

[0143] In one embodiment, a dynamic model of the robotic arm is established, considering its kinematic parameters (such as joint angles, angular velocities, and angular accelerations) and the characteristics of changes in the load's center of gravity position. Using the Lagrange equation or the Newton-Euler equation, the changes in the load's center of gravity position are converted into torques acting on each joint. Specifically, firstly, based on the robotic arm's kinematic relationships, the velocity and acceleration of the load's center of gravity at different times are calculated. Then, combining the changes in load mass and center of gravity position, the dynamic gravity compensation torque value generated by the changes in the load's center of gravity position at each joint is calculated using the dynamic equations. For example, when the robotic arm rotates, the centrifugal force and Coriolis force of the load's center of gravity will generate additional torques on the joints. These torques can be accurately calculated using the dynamic model and included as part of the dynamic gravity compensation torque value. The dynamically calculated gravity compensation torque value can compensate for changes in joint forces caused by changes in the load's center of gravity position in real time, ensuring the accuracy and stability of the robotic arm's movement.

[0144] S503: Add the static gravity compensation torque value and the dynamic gravity compensation torque value to obtain the gravity compensation torque value of each joint of the robotic arm in the next operating cycle.

[0145] In one embodiment, before the start of each operating cycle, the static gravity compensation torque value and the dynamic gravity compensation torque value are calculated based on the load mass distribution characteristics and load center of gravity position change characteristics predicted in the previous cycle. For each joint, the corresponding static gravity compensation torque value and dynamic gravity compensation torque value are simply added together, i.e.: Gravity compensation torque value = Static gravity compensation torque value + Dynamic gravity compensation torque value. For example, for joint i of the robotic arm, the calculated static gravity compensation torque value M_static_i and dynamic gravity compensation torque value M_dynamic_i are added together to obtain M_gravity_compensation_i = M_static_i + M_dynamic_i, which is the gravity compensation torque value of joint i in the next operating cycle. This comprehensively calculated gravity compensation torque value can more comprehensively reflect the influence of load gravity on the joint, providing an accurate basis for subsequent adjustment of the output torque of the drive motors of each joint of the robotic arm, thereby achieving more precise dynamic gravity compensation control.

[0146] In one embodiment, step S503 can be implemented as follows:

[0147] A: The gravity compensation torque value is filtered to generate a gravity compensation torque signal that meets the input requirements of the robotic arm control system.

[0148] In one embodiment, a Butterworth low-pass filter is used to filter the gravity compensation torque value. A suitable cutoff frequency is selected based on the bandwidth requirements and noise frequency characteristics of the robotic arm control system. For example, if the robotic arm control system primarily focuses on low-frequency signals and the noise is mainly concentrated in the high-frequency range, a Butterworth low-pass filter with a cutoff frequency of 10Hz can be selected.

[0149] B: The gravity compensation torque signal is sent to the control system of the robotic arm, instructing the drive motors of each joint of the robotic arm to adjust the output torque according to the gravity compensation torque signal.

[0150] In one embodiment, the gravity compensation torque signal is transmitted to the control system of the robotic arm via a high-speed, reliable communication link. The controller in the control system samples and quantizes the received gravity compensation torque signal, converting it into a digital signal. Then, the controller, according to a preset control algorithm, parses the digital signal into specific control commands for the drive motors of each joint. For example, the controller can use a proportional-integral-derivative (PID) control algorithm to calculate the control quantity that needs adjustment based on the difference between the gravity compensation torque signal and the current motor output torque. By adjusting the input voltage or current of the drive motor, the output torque of the drive motor is changed.

[0151] Accordingly, in order to better implement the above methods, this application also provides a multimodal prediction-based dynamic gravity compensation control system 9 for a robotic arm. Figure 9 As shown, the multimodal prediction-based dynamic gravity compensation control system 80 for the robotic arm includes:

[0152] The acquisition module 801 is used to acquire load information data collected by the robotic arm through multimodal sensors during the current operating cycle. The load information data includes joint torque values ​​recorded by torque sensors, end-effector acceleration values ​​recorded by acceleration sensors, and joint angle values ​​recorded by angle sensors.

[0153] Preprocessing module 802 is used to preprocess the load information data to generate a standardized dataset corresponding to the operating state of the robotic arm. The standardized dataset includes torque change curves, acceleration change curves, and angle change curves in a time series.

[0154] The load prediction module 803 is used to input the standardized dataset into the trained deep learning model and predict the load dynamic characteristics of the robotic arm in the next operating cycle based on the deep learning model. The load dynamic characteristics include load mass distribution characteristics and load center of gravity position change characteristics.

[0155] The compensation calculation module 804 is used to calculate the gravity compensation torque value of each joint of the robotic arm in the next operating cycle based on the prediction result of the load dynamic characteristics. The gravity compensation torque value is obtained by combining the robotic arm dynamic model with the load dynamic characteristics.

[0156] The feedback module 805 is used to feed back the gravity compensation torque value to the control system of the robotic arm in real time, and adjust the output torque of the drive motor of each joint of the robotic arm to realize the dynamic gravity compensation control of the robotic arm in the next operating cycle.

[0157] The implementation details of each module are provided in the preceding method embodiments and will not be repeated here. The technical effects achieved by each module and device are described in the foregoing method embodiments.

[0158] like Figure 10 As shown, this application embodiment also provides a computer device 90, which includes a processor 901 and a memory 902, wherein the memory 902 stores a computer program, and when the computer program is executed by the processor 901, the processor 901 performs the steps of any of the methods described above.

[0159] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A method for dynamic gravity compensation control of a robotic arm based on multimodal prediction, characterized in that, Includes the following steps: The load information data collected by the robotic arm through multimodal sensors during the current operating cycle is acquired. The load information data includes the joint torque value recorded by the torque sensor, the end-effector acceleration value recorded by the acceleration sensor, and the joint angle value recorded by the angle sensor. The load information data is preprocessed to generate a standardized dataset corresponding to the operating state of the robotic arm. The standardized dataset includes torque change curves, acceleration change curves, and angle change curves in a time series. The standardized dataset is input into the trained deep learning model, and the load dynamic characteristics of the robotic arm in the next operating cycle are predicted based on the deep learning model. The load dynamic characteristics include load mass distribution characteristics and load center of gravity position change characteristics. Based on the predicted results of the load dynamic characteristics, the gravity compensation torque values ​​of each joint of the robotic arm in the next operating cycle are calculated. The gravity compensation torque value is obtained by combining the dynamic characteristics of the load with the dynamic model of the robotic arm. The gravity compensation torque value is fed back to the control system of the robotic arm in real time, and the output torque of the drive motor of each joint of the robotic arm is adjusted to realize the dynamic gravity compensation control of the robotic arm in the next operating cycle. Specifically, based on the predicted results of the load dynamic characteristics, the gravity compensation torque values ​​of each joint of the robotic arm in the next operating cycle are calculated, including: Based on the load mass distribution characteristics in the load dynamic characteristics, the equivalent load mass borne by each joint is determined, and the static gravity compensation torque value of each joint of the robotic arm is calculated. Based on the load center of gravity position change characteristics in the load dynamic characteristics, the dynamic gravity compensation torque values ​​of each joint of the robotic arm are calculated. The static gravity compensation torque value and the dynamic gravity compensation torque value are added together to obtain the gravity compensation torque value of each joint of the robotic arm in the next operating cycle.

2. The method according to claim 1, characterized in that, The standardized dataset is input into the trained deep learning model, and the load dynamics of the robotic arm in the next operating cycle are predicted based on the deep learning model, including: Time series features are extracted from the standardized dataset, including the rate of change of torque, the rate of change of acceleration, and the rate of change of angle. The time series features are input into the first convolutional layer of the deep learning model to extract local feature vectors. The local feature vectors are input into the long short-term memory network layer of the deep learning model to generate global feature vectors; Based on the global feature vector, the fully connected layer of the deep learning model outputs the load dynamic characteristics of the robotic arm in the next operating cycle. The load dynamic characteristics include load mass distribution characteristics and load center of gravity position change characteristics.

3. The method according to claim 2, characterized in that, The training process of the deep learning model includes the following steps: The load information data of the robotic arm in multiple historical operating cycles is acquired. The load information data includes the joint torque value recorded by the torque sensor, the end-effector acceleration value recorded by the acceleration sensor, and the joint angle value recorded by the angle sensor. The load information data is labeled to generate a training dataset. The labeled information in the training dataset includes load quality distribution characteristics and load center of gravity position change characteristics. The training dataset is divided into a training subset and a validation subset. The initial deep learning model is trained using the training subset, and the performance of the trained deep learning model is evaluated using the validation subset. Based on the performance evaluation results, the hyperparameters of the deep learning model are adjusted until the prediction error of the deep learning model meets the preset accuracy requirements.

4. The method according to claim 3, characterized in that, The load information data is labeled to generate a training dataset, including: The load physical parameters of the robotic arm in each historical operating cycle are obtained, including the load mass value and the coordinates of the load center of gravity. Based on the physical parameters of the load, the theoretical load dynamic characteristics of the robotic arm in each historical operating cycle are calculated. The theoretical load dynamic characteristics include the theoretical load mass distribution characteristics and the theoretical load center of gravity position change characteristics. The theoretical load dynamic characteristics are used as annotation information and associated with the corresponding load information data to generate a preliminary training dataset. The initial training dataset is subjected to data augmentation processing to generate an expanded training dataset. The data augmentation processing includes data smoothing and noise injection processing. The extended training dataset is divided into a training subset and a validation subset for training and evaluating deep learning models.

5. The method according to claim 1, characterized in that, The gravity compensation torque value is fed back to the control system of the robotic arm in real time, including: The gravity compensation torque value is filtered to generate a gravity compensation torque signal that meets the input requirements of the robotic arm control system. The gravity compensation torque signal is sent to the control system of the robotic arm, instructing the drive motors of each joint of the robotic arm to adjust their output torque according to the gravity compensation torque signal.

6. The method according to any one of claims 1 to 5, characterized in that, The first convolutional layer of the deep learning model extracts local feature vectors through the following steps: Obtain time-series features from a standardized dataset, including the rate of change of torque, the rate of change of acceleration, and the rate of change of angle. The time series features are divided into multiple local time windows, each containing time series data of a fixed length. For each local time window, a sliding convolution operation is performed on the time series data through the convolution kernel of the first convolutional layer to generate a local feature map; The local feature map is subjected to nonlinear activation processing to generate a preliminary local feature vector; The initial local feature vector is input into the pooling layer, and the most representative local feature vector is extracted through max pooling operation, which is then used for global feature generation in the subsequent long short-term memory network layer.

7. The method according to claim 6, characterized in that, The local feature vectors are input into the Long Short-Term Memory (LSTM) network layer of the deep learning model to generate global feature vectors. Includes the following steps: The local feature vectors are arranged in chronological order to form a local feature sequence; The local feature sequence is input into a long short-term memory network layer, and the long-term dependencies in the time series are captured through the memory units of the network layer. At each time step, the state value of the memory cell is updated, and the state value includes the activation values ​​of the input gate, forget gate, and output gate; Based on the state value of the memory unit, the hidden state vector of the current time step is generated; The hidden state vectors of all time steps are concatenated to form a global feature vector, which is used to predict the load dynamic characteristics of the fully connected layer output of the deep learning model.

8. The method according to claim 7, characterized in that, The step of using the fully connected layer of a deep learning model to output the load dynamic characteristics of the robotic arm in the next operating cycle based on the global feature vector includes the following steps: Obtain the global feature vector, which contains long-term dependency information in the time series; The global feature vector is input into the first hidden layer of the fully connected layer, and intermediate feature representations are generated through linear transformation and nonlinear activation functions. The intermediate feature representation is input into the second hidden layer of the fully connected layer to further extract high-level features; Based on the high-level features, load dynamic characteristics prediction results are generated through the output layer of the fully connected layer. The load dynamic characteristics include load mass distribution characteristics and load center of gravity position change characteristics.

9. A dynamic gravity compensation control system for a robotic arm based on multimodal prediction, characterized in that, The system includes: The acquisition module is used to acquire load information data collected by the robotic arm through multimodal sensors during the current operating cycle. The load information data includes joint torque values ​​recorded by torque sensors, end-effector acceleration values ​​recorded by acceleration sensors, and joint angle values ​​recorded by angle sensors. The preprocessing module is used to preprocess the load information data to generate a standardized dataset corresponding to the operating state of the robotic arm. The standardized dataset includes torque change curves, acceleration change curves, and angle change curves over a time series. The load prediction module is used to input the standardized dataset into the trained deep learning model and predict the load dynamic characteristics of the robotic arm in the next operating cycle based on the deep learning model. The load dynamic characteristics include load mass distribution characteristics and load center of gravity position change characteristics. The compensation calculation module is used to calculate the gravity compensation torque value of each joint of the robotic arm in the next operating cycle based on the prediction results of the load dynamic characteristics. The gravity compensation torque value is obtained by combining the robotic arm dynamic model with the load dynamic characteristics. The calculation of the gravity compensation torque value of each joint of the robotic arm in the next operating cycle based on the prediction results of the load dynamic characteristics includes: determining the equivalent load mass borne by each joint based on the load mass distribution characteristics in the load dynamic characteristics, and calculating the static gravity compensation torque value of each joint of the robotic arm; calculating the dynamic gravity compensation torque value of each joint of the robotic arm based on the load center of gravity position change characteristics in the load dynamic characteristics; and adding the static gravity compensation torque value and the dynamic gravity compensation torque value to obtain the gravity compensation torque value of each joint of the robotic arm in the next operating cycle. The feedback module is used to feed back the gravity compensation torque value to the control system of the robotic arm in real time, and adjust the output torque of the drive motors of each joint of the robotic arm to realize the dynamic gravity compensation control of the robotic arm in the next operating cycle.

Images