A method and system for adaptive center of gravity adjustment of a handling robot

Abstract

The present invention discloses a method and system for adaptive center of gravity adjustment of a handling robot, which relates to the field of robot control technology. The method comprises the following steps: using sensors to respectively collect three-dimensional information of an object, contact force distribution information, and contact force information between the robot and the object, integrating the collected information to obtain a multimodal data set and preprocessing the data; using a deep learning algorithm to extract visual feature vectors from three-dimensional image information, and extracting tactile feature vectors from contact force distribution information and contact force information; fusing the extracted visual feature vectors and tactile feature vectors to obtain a multimodal feature set; constructing a motion state model based on a deep learning algorithm, inputting the motion state model using a historical multimodal feature set, and outputting a control strategy; and optimizing the control algorithm and control strategy using real-time feedback information. The present invention proposes an adaptive center of gravity adjustment method by collecting multimodal sensor data and extracting and fusing features in combination with a deep learning algorithm.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a method and system for adaptively adjusting the center of gravity of a handling robot. Background Art

[0002] In recent years, with the rapid development of intelligent manufacturing, logistics automation and service robot technology, handling robots, as an important part of the automation system, have been widely used in many industries. The main task of handling robots is to grasp, carry and place objects accurately and quickly. However, problems such as unstable center of gravity of objects, diversity of grasping and irregularity of object surface often make the stability and efficiency of the handling process face challenges. In order to improve the reliability and flexibility of handling robots, related technologies have gradually developed towards adaptive control and multimodal sensing technology. Adaptive control technology enables robots to adjust their operating strategies in real time according to changes in the environment and objects, while multimodal sensing technology provides robots with rich perceptual inputs by combining visual, tactile and force information, which can help robots perform precise operations in more complex environments.

[0003] Although existing technologies have made significant progress, some shortcomings still exist. Most existing handling robots rely on a single sensor or limited sensor data, making it difficult to fully and accurately perceive the state of an object. For example, although traditional visual sensors can provide information such as the shape and size of an object, they have limited perception capabilities for dynamic characteristics such as the weight and center of gravity of the object. Force sensors can measure contact force, but it is difficult to provide comprehensive information about the object. In addition, existing center of gravity adjustment methods mainly rely on preset control rules or empirical models, lacking sufficient flexibility and adaptability. In particular, when the center of gravity position or contact conditions of an object suddenly change, it is difficult for the robot to quickly adjust its operating strategy, which may cause the object to tilt, fall or be damaged. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a method for adaptively adjusting the center of gravity of a transport robot to solve the problem that most existing transport robots rely on a single sensor or limited sensor data and are difficult to fully and accurately perceive the state of an object.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a method for adaptively adjusting the center of gravity of a transport robot, which comprises:

[0008] Sensors are used to collect the three-dimensional information of the object, the contact force distribution information, and the contact force information between the robot and the object. The collected information is integrated to obtain a multimodal dataset and preprocessed.

[0009] Using deep learning algorithms, visual feature vectors are extracted from 3D image information, and tactile feature vectors are extracted from contact force distribution information and contact force information;

[0010] The extracted visual feature vector and tactile feature vector are fused to obtain a multimodal feature set;

[0011] Build a motion state model based on deep learning algorithm, use historical multimodal feature set to input the motion state model, and output control strategy;

[0012] Utilize real-time feedback information to optimize control algorithms and control strategies.

[0013] As a preferred embodiment of the method for adaptively adjusting the center of gravity of a handling robot according to the present invention, the method comprises the following steps: using sensors to respectively collect three-dimensional information of an object, contact force distribution information, and contact force information between the robot and the object; integrating the collected information to obtain a multimodal data set; and preprocessing the data set.

[0014] The RGB-D camera is used to obtain three-dimensional image information of the object. At the same time, a pressure sensor array is arranged on the robot's end effector to collect contact force distribution information. In addition, torque sensors are installed on the robot's joint actuators to measure the contact force information between the robot and the object.

[0015] Integrating three different types of data into a multimodal dataset , the expression is:

[0016] ;

[0017] in, Indicates time 3D image information, Indicates time Contact force distribution information, Indicates time contact force information.

[0018] As a preferred solution of the method for adaptively adjusting the center of gravity of a handling robot according to the present invention, wherein: The time series data of one dimension selected in the ;

[0019] Apply continuous wavelet transform (CWT) to the original signal to decompose;

[0020] Through continuous wavelet transform CWT, a series of different scales are obtained The detail coefficient and approximation coefficient under ;

[0021] Set the noise threshold. When the detail coefficient at a certain scale is less than the noise threshold, this part is considered to be noise and is set to zero.

[0022] The signal is reconstructed using the denoised detail coefficients and approximate coefficients. The expression is:

[0023] ;

[0024] in, represents the reconstructed signal, represents the inverse continuous wavelet transform, represents the detail coefficient after noise threshold processing, represents the approximate coefficient, An index variable representing the scale, Indicates the maximum scale index.

[0025] As a preferred solution of the method for adaptive center of gravity adjustment of the handling robot of the present invention, the convolutional neural network CNN is used to obtain the three-dimensional image information. Extract visual feature vectors ;

[0026] Using long short-term memory network LSTM to obtain contact force distribution information and contact force information Extract tactile feature vectors .

[0027] As a preferred solution of the method for adaptively adjusting the center of gravity of a handling robot according to the present invention, the method of fusing the extracted visual feature vector and tactile feature vector to obtain a multimodal feature set comprises the following specific steps:

[0028] The weighted average method is used to fuse the visual feature vector and the tactile feature vector to obtain a multimodal feature set. , the expression is:

[0029] ;

[0030] in, and Represent visual feature vectors and tactile feature vector The weight coefficient of Indicates time, Indicates that the item is at time The visual feature vector of Indicates that the item is at time The tactile feature vector of .

[0031] As a preferred solution of the method for adaptive center of gravity adjustment of the handling robot of the present invention, wherein: the motion state model is constructed based on the deep learning algorithm, the motion state model is input using the historical multimodal feature set, and the control strategy is output, the specific steps are:

[0032] The motion state model is constructed based on the combination of long short-term memory (LSTM) and multi-layer perceptron (MLP). The LSTM is used to process the dynamic characteristics of time series data and can capture the dependencies of multimodal features on the time axis. The MLP is used to process the fused multimodal features and learn the complex mapping relationships between the features.

[0033] Collect historical multimodal feature sets , and input it into the motion state model, and output the control strategy, the expression is:

[0034] ;

[0035] in, Indicates the robot at time The control strategies adopted, represents the activation function Sigmoid, represents the weight matrix of the multilayer perceptron MLP output layer, Represents the bias term of the multi-layer perceptron MLP output layer;

[0036] The control strategy that will be output It is transmitted to the robot actuator to guide the robot's movement and posture adjustment.

[0037] As a preferred solution of the method for adaptively adjusting the center of gravity of a handling robot according to the present invention, the method of utilizing real-time feedback information to optimize the control algorithm and control strategy comprises the following specific steps:

[0038] When the robot performs a task, it collects real-time feedback information based on sensors;

[0039] The original signal collected by the sensor , the original signal After a series of preprocessing steps, the current state of the robot is finally obtained ;

[0040] Based on the current state of the robot and control strategies , the quality of the control strategy is evaluated by the Q-value function, the goal is to maximize the long-term return, and the Q-learning update formula is:

[0041] ;

[0042] in, Indicates the current status and control strategies The Q value under represents the learning rate, Indicates time Instant rewards, represents the discount factor, represents the updated control strategy, Indicates that the selection is in state The control strategy that can obtain the maximum future reward is: Indicates the robot at time status, Indicates the robot at time the control strategies adopted;

[0043] according to The control strategy that produces the maximum Q value in a given state is selected to guide the robot's actions.

[0044] In a second aspect, the present invention provides a transport robot adaptive center of gravity adjustment system, comprising: an acquisition module, a feature extraction module, a strategy generation module, a feedback module, and an optimization module:

[0045] The acquisition module is responsible for collecting and pre-processing the three-dimensional information of the object, the contact force distribution information, and the contact force information between the robot and the object;

[0046] The feature extraction module is responsible for extracting visual feature vectors from 3D image data and tactile feature vectors from contact force distribution and contact force information. These features are then fused through weighted averaging to generate a final multimodal feature set.

[0047] The strategy generation module is used to construct a robot motion state model based on the extracted multimodal feature set and predict the robot's motion state;

[0048] The feedback module is used to collect and analyze feedback information during the robot's execution process in real time and optimize the control strategy;

[0049] The optimization module is responsible for continuously collecting new sensor data, updating historical data sets, and adjusting the motion state model through optimization algorithms.

[0050] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the adaptive center of gravity adjustment method for a handling robot as described in the first aspect of the present invention is implemented.

[0051] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the method for adaptive center of gravity adjustment of a handling robot as described in the first aspect of the present invention is implemented.

[0052] The beneficial effects of the present invention are as follows: the present invention collects multimodal sensor data and combines it with a deep learning algorithm for feature extraction and fusion, and proposes an adaptive center of gravity adjustment method. By using an RGB-D camera, a pressure sensor array, and a torque sensor, the interaction information between the object and the robot can be comprehensively and accurately acquired, thereby providing a reliable basis for subsequent control. Wavelet transform is used to denoise and enhance the signal to ensure high data quality. The convolutional neural network (CNN) and the long short-term memory (LSTM) network are combined to effectively extract visual and tactile features, and then a multimodal feature set is obtained through weighted average fusion to accurately reflect the state of the object. Based on these features, the constructed motion state model can predict and adaptively adjust the robot control strategy. Real-time feedback combined with the Q-learning optimization algorithm further improves the robot's decision-making ability and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 This is a flow chart of the method for adaptively adjusting the center of gravity of the transport robot in Example 1.

[0055] Figure 2 Schematic diagram of the adaptive center of gravity adjustment system of the transport robot in Example 1. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0057] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0058] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0059] Example 1, with reference to Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides a method for adaptively adjusting the center of gravity of a handling robot, comprising the following steps:

[0060] S1. Use sensors to collect the three-dimensional information of the object, the contact force distribution information, and the contact force information between the robot and the object, integrate the collected information to obtain a multimodal dataset, and preprocess it;

[0061] An RGB-D camera captures three-dimensional images of objects, providing key visual features about their shape and size. A pressure sensor array is placed on the robot's end-effector to collect contact force distribution information, which reflects the pressure at the contact point between the object's surface and the robot. Furthermore, torque sensors are installed on the robot's joint actuators to measure the contact force between the robot and the object, which is crucial for understanding the object's weight distribution and grasping stability.

[0062] Integrating three different types of data into a multimodal dataset , the expression is:

[0063] ;

[0064] in, Indicates time 3D image information, Indicates time Contact force distribution information, Indicates time Contact force information;

[0065] The 3D image information includes the distance and color values of each pixel point, the contact force distribution information includes the pressure value of each sensor unit, and the contact force information between the robot and the object includes the force components along the X-axis, Y-axis, and Z-axis;

[0066] From multimodal datasets The time series data of a dimension selected in is used to represent the change of the physical quantity under this dimension over time, and the original signal is obtained ;

[0067] Considering that the data provided by different sensors contain noise or missing values, an effective algorithm is needed for denoising and enhancement. For this step, a method based on wavelet transform is selected as the optimal solution for signal processing. Wavelet transform can effectively separate the high-frequency noise components in the signal while retaining the effective low-frequency information, thereby improving the accuracy of subsequent analysis.

[0068] Specifically, the continuous wavelet transform (CWT) is applied to the original signal Decomposed, the expression is:

[0069] ;

[0070] in, Represents the original signal after wavelet transform At different scales The transformation result is represents the index of the scale, represents the integration variable, represents the mother wavelet function In scale The complex conjugate of represents differential elements;

[0071] Through continuous wavelet transform CWT, a series of different scales are obtained The detail coefficient and approximation coefficient under ;

[0072] Set a noise threshold. When the detail coefficient at a certain scale is less than the noise threshold, this part is considered to be noise and is set to zero. This hard threshold method can effectively remove high-frequency noise while retaining the effective low-frequency information.

[0073] The signal is reconstructed using the denoised detail coefficients and approximate coefficients to achieve the final denoising effect. The expression is:

[0074] ;

[0075] in, represents the reconstructed signal, represents the inverse continuous wavelet transform, represents the detail coefficient after noise threshold processing, represents the approximate coefficient, An index variable representing the scale, Indicates the maximum scale index.

[0076] S2. Using deep learning algorithms, extract visual feature vectors from 3D image information; extract tactile feature vectors from contact force distribution information and contact force information;

[0077] Using convolutional neural network CNN from 3D image information Extract visual feature vectors ;

[0078] By extracting visual features, the geometric model of the object is established, providing accurate spatial information for center of gravity adjustment. This solves the problem that traditional image feature processing methods cannot adequately reflect the morphology of complex objects. CNN can analyze layer by layer from local details to the overall structure, avoiding the limitations of manually set features. This improves the recognition accuracy and adaptability of complex objects and provides more reliable input data for subsequent posture adjustment and force control strategies.

[0079] Using long short-term memory network LSTM to obtain contact force distribution information and contact force information Extract tactile feature vectors ;

[0080] By dynamically analyzing tactile data to establish an object force model, the robot can adjust its grasping strategy and posture control. This solves the problem that traditional force control systems can only process static data and have difficulty capturing dynamic force changes. LSTM uses a memory and forgetting mechanism to analyze the connection between historical data and current status, providing more flexible force perception capabilities. This enhances the robot's perception of dynamic force changes, improves its adaptability and stability to complex handling tasks, and helps maintain balance and grasping stability with irregular objects or in dynamic environments.

[0081] Through the step-by-step extraction of visual and tactile features and multi-dimensional data processing, the present invention has significant advantages in adjusting the center of gravity of the robot. Visual features provide geometric information support, while tactile features capture dynamic force changes. The combination of the two can form a more comprehensive description of the object state. This multimodal perception solution overcomes the defect that single sensor data is not enough to accurately reflect complex scenes, and significantly improves the robot's environmental adaptability and task completion rate. In particular, in scenarios requiring high-precision posture adjustment (such as precision assembly or flexible handling), the present invention provides a more reliable and intelligent solution.

[0082] S3, fusing the extracted visual feature vector and tactile feature vector to obtain a multimodal feature set;

[0083] The weighted average method is used to fuse the visual feature vector and the tactile feature vector to obtain a multimodal feature set. , the expression is:

[0084] ;

[0085] in, and Represent visual feature vectors and tactile feature vector The weight coefficient of Indicates time, Indicates that the item is at time The visual feature vector of Indicates that the item is at time The tactile feature vector of

[0086] The multimodal feature fusion method proposed in this paper overcomes the limited expressive power of single-modal features by combining visual and tactile features in a weighted manner. In traditional technologies, visual or tactile features are usually processed separately, which cannot provide a description of the global state of the object, resulting in unstable or inaccurate control strategies.

[0087] By using multimodal feature fusion and integrating visual and tactile information, the robot is provided with a more complete object perception capability, which solves the limitation of single sensor data expression, especially the response delay of visual features to dynamic environments and the lack of global information of tactile features. Combining the spatial distribution advantage of visual information with the dynamic feedback advantage of tactile information improves the robot's environmental adaptability and control accuracy;

[0088] Step S3 fuses visual and tactile features through weighted averaging to construct a multimodal feature set. , plays a core role in the robot control algorithm. This design not only improves the data expression ability and adaptability, but also introduces dynamic time parameters and weight coefficient and , which solves the problem of ignoring the importance of different features in traditional methods. Its innovation lies in making full use of the complementarity of visual and tactile information, enabling the robot to maintain high-precision control and adaptive adjustment in dynamic and complex environments to meet practical application needs.

[0089] S4. Build a motion state model based on a deep learning algorithm, use the historical multimodal feature set as input to the motion state model, and output a control strategy;

[0090] The motion state model is constructed by combining a long short-term memory (LSTM) network and a multi-layer perceptron (MLP). The LSTM network is used to process the dynamic characteristics of time series data and can capture the dependencies of multimodal features on the time axis. The MLP is used to process the fused multimodal features and learn the complex mapping relationships between them. In this way, the motion state model can predict the robot's current motion state based on historical data, thereby achieving adaptive adjustment.

[0091] LSTM is suitable for processing time series data and can capture the impact of past information on current predictions. It is particularly suitable for robot kinematics and dynamic control problems.

[0092] MLP can learn multiple input features in parallel and is good at extracting high-dimensional information from different features. The advantages of MLP are more obvious when we need to process fused visual and tactile features.

[0093] Collect historical multimodal feature sets , and input it into the motion state model, and output the control strategy, the expression is:

[0094] ;

[0095] in, , represents the activation function Sigmoid, represents the weight matrix of the multilayer perceptron MLP output layer, Represents the bias term of the multilayer perceptron MLP output layer;

[0096] The control strategy that will be output The robot actuator is passed to guide the robot's movement and posture adjustment. During the execution process, the robot will correct its motion state according to real-time feedback (such as the output of pressure sensors and torque sensors) and update the historical feature set. , continuously optimize the motion state model;

[0097] Control strategy The range of is [0,1], which represents the probability of the control strategy. For example, the probability of the control strategy pointing to a certain posture adjustment, and the value in the range will indicate the probability of performing the action;

[0098] Weight Matrix It is learned through the back propagation algorithm. Specifically, the motion state model is based on the training data (i.e. the historical multimodal feature set and corresponding control strategies Constant Adjustment And other network parameters, during the training process, the optimization algorithm Adam is used to minimize the loss function, thereby updating and other parameters.

[0099] S5. Utilize real-time feedback information to optimize control algorithms and control strategies;

[0100] When the robot performs a task, it collects real-time feedback information based on sensors, including real-time contact force information, real-time contact force distribution information, and real-time motion state information;

[0101] Based on the collected real-time feedback information, the motion state model is updated using Q-learning in reinforcement learning. Q-learning can evaluate the effectiveness of the current control strategy and optimize future decisions through real-time feedback information.

[0102] The original signal collected by the sensor , the original signal After a series of preprocessing steps (such as denoising, wavelet transform, CNN and LSTM), the current state of the robot is finally obtained , including visual feature vectors, tactile feature vectors and multimodal feature sets;

[0103] Based on the current state of the robot and control strategies , the quality of the control strategy is evaluated by the Q-value function, the goal is to maximize the long-term return (ie, cumulative reward), and the Q-learning update formula is:

[0104] ;

[0105] in, Indicates the current status and control strategies The Q value under represents the learning rate, Indicates time The immediate reward is the feedback value obtained by the robot after executing the control strategy in this state. represents the discount factor, represents the updated control strategy, Indicates that the selection is in state The control strategy that can obtain the maximum future reward is: Indicates the robot at time status, Indicates the robot at time the control strategies adopted;

[0106] according to The control strategy that produces the maximum Q value in a given state is selected to guide the robot's actions.

[0107] This embodiment also provides a transport robot adaptive center of gravity adjustment system, including: an acquisition module, a feature extraction module, a strategy generation module, a feedback module and an optimization module:

[0108] The acquisition module is responsible for collecting and preprocessing the three-dimensional information of the object, the contact force distribution information, and the contact force information between the robot and the object;

[0109] The feature extraction module is responsible for extracting visual feature vectors from 3D image data and tactile feature vectors from contact force distribution and contact force information. These features are then fused through weighted averaging to generate the final multimodal feature set.

[0110] Strategy generation module, used to build a robot motion state model based on the extracted multimodal feature set and predict the robot's motion state;

[0111] Feedback module, used to collect and analyze feedback information during the robot's execution in real time and optimize the control strategy;

[0112] The optimization module is responsible for continuously collecting new sensor data, updating historical data sets, and adjusting the motion state model through optimization algorithms.

[0113] This embodiment also provides a computer device, which is suitable for the case of the adaptive center of gravity adjustment method of the handling robot, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the adaptive center of gravity adjustment method of the handling robot proposed in the above embodiment.

[0114] The computer device may be a terminal, comprising a processor, memory, a communication interface, a display, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and computer programs. The internal memory provides an environment for the operating system and computer programs stored in the non-volatile storage media. The communication interface of the computer device is used to communicate with external terminals via wired or wireless communication. Wireless communication may be achieved via Wi-Fi, a carrier network, NFC (near-field communication), or other technologies. The display of the computer device may be a liquid crystal display or an electronic ink display. The input device may be a touchscreen overlay on the display, buttons, a trackball, or a touchpad on the computer device housing, or an external keyboard, touchpad, or mouse.

[0115] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for implementing the adaptive center of gravity adjustment of a handling robot as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0116] In summary, the present invention collects multimodal sensor data and combines it with a deep learning algorithm for feature extraction and fusion, and proposes an adaptive center of gravity adjustment method. By using an RGB-D camera, a pressure sensor array, and a torque sensor, the interaction information between the object and the robot can be comprehensively and accurately acquired, thereby providing a reliable basis for subsequent control. Wavelet transform is used to denoise and enhance the signal to ensure high data quality. The convolutional neural network (CNN) and the long short-term memory (LSTM) network are combined to effectively extract visual and tactile features. Then, through weighted average fusion, a multimodal feature set is obtained to accurately reflect the state of the object. Based on these features, the constructed motion state model can predict and adaptively adjust the robot control strategy. Real-time feedback combined with the Q-learning optimization algorithm further improves the robot's decision-making ability and stability.

[0117] Example 2, referring to Table 1, is the second embodiment of the present invention. In order to further verify the technical solution of the present invention, experimental simulation data of the adaptive center of gravity adjustment method of the handling robot are provided.

[0118] This experiment aims to verify the effectiveness and superiority of the robot control method based on multimodal feature fusion in complex object grasping and posture adjustment. The experimental platform includes:

[0119] Robotic system: A six-degree-of-freedom robotic arm equipped with an end effector and force sensor.

[0120] Visual sensor: A depth camera used to capture three-dimensional image information of objects.

[0121] Tactile sensor: An array of pressure sensors mounted on the end of the robotic arm to measure contact force distribution.

[0122] Data processing: Use high-performance processors.

[0123] The experimental subjects were objects of three different shapes and surfaces, including a regular rectangular object (Object A), a cylindrical object (Object B), and an object with an irregular surface (Object C). The experimental process was as follows:

[0124] Data collection phase:

[0125] Use RGB-D cameras to capture three-dimensional image information of objects , with a resolution of 1280x720 and a frame rate of 30fps, recording object size and shape characteristics.

[0126] Use pressure sensors to record contact force distribution information , the sampling rate is 200 Hz, and the pressure distribution between the actuator and the object is measured.

[0127] The torque sensor collects the contact force information between the robot and the object , with a resolution of 0.1N.

[0128] All collected data are synchronized and a multimodal dataset is constructed .

[0129] Data preprocessing stage:

[0130] The collected data were denoised and the continuous wavelet transform (CWT) was used to decompose and reconstruct the signal. The threshold was set to 1.5σ (standard deviation).

[0131] The denoised signals are resynthesized to ensure a high signal-to-noise ratio for the visual and tactile data.

[0132] Feature extraction stage:

[0133] Visual features are extracted through convolutional neural network (CNN) and output as visual feature vectors .

[0134] Tactile features are extracted through the long short-term memory network LSTM, and the tactile feature vector is output. .

[0135] Multimodal feature fusion stage:

[0136] According to the expression , the visual and tactile features are integrated by weighted averaging method, and the weights are set =0.6, =0.4 to enhance the responsiveness of dynamic information during the crawling process.

[0137] Control strategy generation and execution phase:

[0138] Input fusion feature set To LSTM-MLP model, generate control strategy .

[0139] The control strategy is optimized in real time based on Q-learning, and the grasping posture is corrected by the immediate reward r(t).

[0140] The details are shown in Table 1 below:

[0141] Table 1 Experimental data and comparative analysis

[0142]

[0143] Experimental data show that the proposed method outperforms existing methods in key indicators such as grasping success rate, number of posture adjustments, average force error, and object stability score. The following is a detailed analysis of each indicator:

[0144] Grasping success rate: The success rate of our method is generally 10-23 percentage points higher than that of existing methods, showing a significant advantage in particular for objects C with irregular surfaces, verifying the adaptability of multimodal feature fusion to complex objects.

[0145] Number of posture adjustments: Existing methods require multiple adjustments to the grasping posture to achieve a stable state. However, this invention reduces the number of adjustments to 3-4 times based on real-time feedback and optimized control strategies of multimodal data, thereby improving execution efficiency.

[0146] Average force error: The average error of the present invention is controlled between 0.12-0.20N, which is significantly lower than the 0.45-0.72N of the existing method. This shows that the fusion of visual and tactile features effectively improves the perception accuracy of the force state of the object;

[0147] Data processing and execution time: The present invention reduces the data processing and execution phases by 20-35ms and 30-45ms respectively, mainly due to the efficient denoising of wavelet transform and the fast feature extraction of deep learning model;

[0148] Object stability score: The score of the proposed method is above 9.0, far exceeding the average score of 6.5 of existing methods, further proving that multimodal feature fusion can improve the robot's grasping stability.

[0149] It can be seen from the experimental results that the present invention has significant innovations in the field of complex object grasping and posture control through multimodal data fusion, feature extraction and dynamic control strategy optimization. Compared with traditional single sensors or independent processing modes, the present invention fully utilizes the complementarity of visual and tactile features, enabling the robot to exhibit higher robustness and adaptability in dynamic environments. Especially in the grasping of irregular or complex-surface objects, traditional methods are prone to frequent or failed posture adjustments, while the present invention significantly improves control accuracy and task success rate through real-time feedback and optimization mechanisms. Therefore, the invention provides a more efficient and reliable solution for application scenarios such as automated production lines, intelligent handling and precision assembly.

[0150] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for adaptively adjusting the center of gravity of a handling robot, characterized by: include: Sensors are used to collect the three-dimensional information of the object, the contact force distribution information, and the contact force information between the robot and the object. The collected information is integrated to obtain a multimodal dataset and preprocessed. Using deep learning algorithms, we extract visual feature vectors from 3D image information and tactile feature vectors from contact force distribution and contact force information. Specifically: Using convolutional neural network CNN from 3D image information Extract visual feature vectors ; Using long short-term memory network LSTM to obtain contact force distribution information and contact force information Extract tactile feature vectors ; The extracted visual feature vector and tactile feature vector are fused to obtain a multimodal feature set, specifically: The weighted average method is used to fuse the visual feature vector and the tactile feature vector to obtain a multimodal feature set. , the expression is: ； in, and Represent visual feature vectors and tactile feature vector The weight coefficient of Indicates time, Indicates that the item is at time The visual feature vector of Indicates that the item is at time The tactile feature vector of The motion state model is constructed based on the deep learning algorithm. The historical multimodal feature set is used as input to the motion state model and the control strategy is output. Specifically: The motion state model is constructed based on the combination of long short-term memory network LSTM and multi-layer perceptron MLP; Collect historical multimodal feature sets , and input it into the motion state model, and output the control strategy, the expression is: ； in, Indicates the robot at time The control strategies adopted, represents the activation function Sigmoid, represents the weight matrix of the multilayer perceptron MLP output layer, Represents the bias term of the multilayer perceptron MLP output layer; The control strategy that will be output Transmitted to the robot actuator to guide the robot's movement and posture adjustment; Utilize real-time feedback information to optimize control algorithms and control strategies, specifically: When the robot performs a task, it collects real-time feedback information based on sensors; The original signal collected by the sensor , the original signal After a series of preprocessing steps, the current state of the robot is finally obtained ; Based on the current state of the robot and control strategies , the quality of the control strategy is evaluated by the Q-value function, the goal is to maximize the long-term return, and the Q-learning update formula is: ； in, Indicates the current status and control strategies The Q value under represents the learning rate, Indicates time Instant rewards, represents the discount factor, represents the updated control strategy, Indicates that the selection is in state The control strategy that can obtain the maximum future reward is: Indicates the robot at time status, Indicates the robot at time the control strategies adopted; according to The control strategy that produces the maximum Q value in a given state is selected to guide the robot's actions.

2. The method for adaptively adjusting the center of gravity of a handling robot according to claim 1, wherein: The sensors are used to respectively collect the three-dimensional information of the object, the contact force distribution information, and the contact force information between the robot and the object, and the collected information is integrated to obtain a multimodal data set and preprocessed. The specific steps are: The RGB-D camera is used to obtain three-dimensional image information of the object. At the same time, a pressure sensor array is arranged on the robot's end effector to collect contact force distribution information. In addition, torque sensors are installed on the robot's joint actuators to measure the contact force information between the robot and the object. Integrating three different types of data into a multimodal dataset , the expression is: ； in, Indicates time 3D image information, Indicates time Contact force distribution information, Indicates time contact force information.

3. The method for adaptively adjusting the center of gravity of a handling robot according to claim 2, wherein: The pre-processing comprises the following specific steps: From multimodal datasets The time series data of one dimension selected in the ; Apply continuous wavelet transform (CWT) to the original signal to decompose; Through continuous wavelet transform CWT, a series of different scales are obtained The detail coefficient and approximation coefficient under ; Set the noise threshold. When the detail coefficient at a certain scale is less than the noise threshold, this part is considered to be noise and is set to zero. The signal is reconstructed using the denoised detail coefficients and approximate coefficients. The expression is: ； in, represents the reconstructed signal, represents the inverse continuous wavelet transform, represents the detail coefficient after noise threshold processing, represents the approximate coefficient, An index variable representing the scale, Indicates the maximum scale index.

4. A transport robot adaptive center of gravity adjustment system, based on the transport robot adaptive center of gravity adjustment method according to any one of claims 1 to 3, characterized in that: include: Acquisition module, feature extraction module, strategy generation module, feedback module and optimization module: The acquisition module is responsible for collecting and pre-processing the three-dimensional information of the object, the contact force distribution information, and the contact force information between the robot and the object; The feature extraction module is responsible for extracting visual feature vectors from 3D image data and tactile feature vectors from contact force distribution and contact force information. These features are then fused through weighted averaging to generate a final multimodal feature set. The strategy generation module is used to construct a robot motion state model based on the extracted multimodal feature set and predict the robot's motion state; The feedback module is used to collect and analyze feedback information during the robot's execution process in real time and optimize the control strategy; The optimization module is responsible for continuously collecting new sensor data, updating historical data sets, and adjusting the motion state model through optimization algorithms.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for adaptively adjusting the center of gravity of a handling robot according to any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for adaptively adjusting the center of gravity of a handling robot according to any one of claims 1 to 3 are implemented.

Images