Multi-modal sensing integrated robot cooperative operation method

Abstract

The invention discloses a multi-modal sensing integrated robot cooperative operation method, and relates to the technical field of robot control, and the method comprises the following steps: S1, data collection: employing a visual sensor, a touch sensor, an auditory sensor and an inertial sensor to form a sensor array, multi-modal sensing data of a robot working environment are collected and obtained in real time, wherein the multi-modal sensing data comprise an image sequence, a contact force distribution matrix, a voiceprint spectrogram and robot six-degree-of-freedom pose information. According to the multi-modal sensing integrated robot cooperative operation method, comprehensive and real-time sensing of the robot operation environment is achieved by integrating visual, tactile, auditory and inertial sensor data, and the robot cooperative operation method is used for adapting to constantly changing environment and task requirements by updating an environment sensing model and setting feedback adjustment strategies in real time, so that the robot cooperative operation efficiency is improved. And thus, the environment perception capability, decision accuracy and cooperative operation capability of the multi-modal sensing integrated robot are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a multi-modal sensing integrated robot collaborative operation method. Background Art

[0002] Robot control technology refers to applying input control to the robot's drive device to complete the set tasks. It covers multiple aspects of the robot's perception, decision-making, execution, etc. It is the key to the robot's ability to perform tasks autonomously or according to programs. By using a variety of sensors, it can collect multimodal data of the robot's operating environment in real time, providing a basis for the robot's decision-making and execution.

[0003] The defects of existing robot control technology are:

[0004] 1. Patent document CN115151366B discloses a robot control device, "providing a robot control device capable of suppressing the stop of operation caused by contact misdetection. A robot control device is used to control a robot that performs contact welding on an object to be welded, and the robot control device comprises: an action stop unit, which stops the action of the robot when it detects that the robot is subjected to an external force above a threshold; an indication unit, which indicates to a welding power supply device to start welding; and a detection sensitivity adjustment unit, which reduces the sensitivity of detecting external force in the action stop unit during the period from the time when the indication unit indicates to the welding power supply device to start welding to the time when a prescribed standby time has passed", but the existing robot control device has the problem of controlling the robot only through data obtained by a single sensor, and has the problem of poor control decision-making and unstable operation;

[0005] 2. Patent document CN108724223B discloses a robot control device. "The present invention discloses a robot control device. According to one embodiment of the present invention, the robot control device comprises: a robot controller connected to a robot to control the action of the connected robot; and a teach pendant connected to the robot controller to generate a control signal for controlling the action of the robot according to the user's operation and transmit it to the robot controller, and the robot controller comprises: a control unit to process a first control signal not related to safety input from the teach pendant and output a control instruction; a safety PLC (Programmable Logic Control) to process a second control signal related to safety input from the teach pendant through soft wiring logic and output a safety signal; and a safety unit to process a safety signal input from the safety PLC or a third control signal related to safety input from the teach pendant through hard wiring logic and output an emergency stop instruction for controlling the robot." However, the existing robot control device cannot collect and use data from multiple sensors at the same time, resulting in the technical problem that the existing robot control device has poor perception ability when facing a complex working environment.

[0006] 3. Patent document CN105690382A discloses a robot control device. "The present invention provides a robot control device, which can flexibly change the setting method according to the setting environment. The robot control device is a robot control device connected to a robot and controls the above robot. It has a shell and a display unit. The above display unit can move relative to the above shell and display information." However, when a single sensor of the existing robot control device is damaged during use, it leads to a technical problem that the robot's actions cannot be controlled. Summary of the invention

[0007] The purpose of the present invention is to provide a multi-modal sensing integrated robot collaborative operation method to solve the technical problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: a multi-modal sensing integrated robot collaborative operation method, comprising the following steps:

[0009] Step S1, data acquisition: by using visual sensors, tactile sensors, auditory sensors and inertial sensors to form a sensor array, multimodal sensor data of the robot's working environment is acquired in real time, including image sequences, contact force distribution matrices, voiceprint spectrograms and robot six-degree-of-freedom posture information;

[0010] Step S2, data preprocessing: preprocess the collected multimodal data, including denoising, filtering, spatiotemporal calibration and fault-tolerant processing, to ensure the quality and consistency of the data. When any sensor fails or is damaged, other sensors can continue to work and provide effective data support to ensure that the robot can continue to perform tasks.

[0011] Step S3, data fusion: using a data fusion algorithm to fuse the pre-processed multimodal data to generate a comprehensive environment perception model, which can more comprehensively reflect the actual situation of the robot's environment, including the location, shape, material and dynamic changes of objects;

[0012] Step S4, feature extraction and recognition: extract key features based on the fused multimodal data, and use machine learning or deep learning algorithms to perform pattern recognition to identify objects, people or sound data in the environment, including the category of objects, the actions of people and the source of sounds;

[0013] Step S5, decision making: according to the recognition results, combined with the robot's task objectives and current state, make collaborative operation decisions, including path planning, action selection and interaction strategies, and judge in real time whether the decision can be completed accurately and the safety of the robot during movement, to ensure the reliability and safety of the operation, and during the movement, detect the path environment to determine whether there are potential dangers in the planned path;

[0014] Step S6, collaborative execution: The robot collaborates with other robots or humans to perform corresponding tasks based on the decision results, updates the environmental perception model in real time, and adjusts the decision and execution strategies in time according to the feedback adjustment steps to adapt to the ever-changing environment and task requirements.

[0015] Preferably, the step S1 collects multimodal data in real time in a smart warehouse logistics environment (floor height 3-5 meters, channel width 0.8-1.2 meters, artificial lighting 300-500 lux, background noise 45-60 dB) by integrating visual sensors, tactile sensors, auditory sensors and inertial sensors;

[0016] During the data collection process, the visual sensor uses an Intel RealSense D455 RGB-D camera to collect RGB-D image sequences at a resolution of 1280×720 and 30fps, with a depth measurement range of 0.2-10m and an accuracy of ±2%;

[0017] The tactile sensor uses a 32×32 taxel pressure distribution array with a range of 0-20N and a sampling rate of 100Hz, and outputs a contact force distribution matrix and a pressure gradient field;

[0018] The auditory sensor uses an 8-element circular microphone array (frequency response 20Hz-20kHz) to generate MFCC voiceprint features and sound source azimuth data;

[0019] The inertial sensor uses MPU-6050 IMU, which outputs three-axis acceleration (±2g), three-axis angular velocity (±250° / s), and the data frequency is 100Hz.

[0020] Preferably, in step S2, wavelet threshold denoising (threshold coefficient 0.8) and Kalman filtering (process noise covariance 0.01) are used in the data preprocessing process;

[0021] Filtering, used to remove unnecessary fluctuations and jitters;

[0022] Space-time calibration: μs-level time synchronization is achieved through hardware trigger pulses, and the spatial grid calibration method is used to compensate for sensor position deviation;

[0023] Fault-tolerance processing: Design a redundant sensor voting mechanism to ensure that system availability is ≥ 95% when a single sensor fails.

[0024] Preferably, in the step S3, in the data fusion step, an adaptive weighted fusion algorithm based on information entropy is adopted, and the formula is expressed as:

[0025] ;

[0026] Among them, H i is the information entropy of the i-th sensor, and the calculation formula is:

[0027] ;

[0028] The weight distribution strategy is:

[0029] Visual weight: Base 0.4 → Increased to 0.6 for moving object detection;

[0030] Haptic weight: Base 0.15 → jumps to 0.35 when contact force changes suddenly;

[0031] Hearing weight: Base 0.05 → Increased to 0.2 when abnormal noise event is triggered;

[0032] Inertia weight: constant 0.2 as a reference;

[0033] Among them, W i represents the weight of the i-th data, H i represents the information entropy of the i-th sensor, H j represents the information entropy of the jth sensor, represents an exponential function with the natural constant e as the base, in which the information entropy value is converted into a non-negative weight coefficient. The larger the entropy value, the The smaller it is, the more adaptive weight adjustment can be achieved. The summation symbol is used to calculate the sum of all items from j=1 to j=n, indicating the traversal calculation of n sensor data. represents the probability value of the kth feature of the i-th sensor, represents the number of feature dimensions, It is a logarithmic function used to calculate information entropy.

[0034] Preferably, in step S4, in the feature extraction and recognition step, a multimodal feature space is constructed; visual features: ORB feature points + depth gradient histogram, tactile features: pressure field principal component analysis (PCA) dimensionality reduction, auditory features: MFCC + voiceprint zero-crossing rate, inertial features: motion trajectory frequency domain analysis, and an improved YOLOv5 network is used for multimodal target detection, achieving mAP@0.5≥92% on a custom warehouse data set.

[0035] Preferably, the decision making in step S5 includes designing a two-layer decision architecture:

[0036] Path planning layer: The RRT*-Connect algorithm is used to generate obstacle avoidance paths, and the path cost function is adjusted in real time through tactile feedback;

[0037] Action selection layer: Based on the DQN reinforcement learning framework, multimodal fusion features are used as state input to train a policy network with a grasping success rate of ≥ 98%.

[0038] Preferably, in step S6, in the collaborative execution step, a collaborative mechanism based on digital twins is implemented:

[0039] Establish a virtual warehousing environment and synchronize physical world data in real time (delay <50ms);

[0040] The MQTT protocol is used to realize status sharing between robots, and the communication bandwidth requirement is <1Mbps;

[0041] Design a dynamic task allocation algorithm to adjust task allocation in real time according to the load status (CPU / memory usage) of each robot.

[0042] Preferably, the feedback adjustment step in step S6 is used to adjust the environmental perception model, decision-making and execution strategy in real time according to the task execution status and environmental changes during the collaborative execution process to improve the efficiency and accuracy of the collaborative operation.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. The present invention realizes comprehensive and real-time perception of the robot's working environment through the robot collaborative operation method of multimodal sensing integration by integrating visual, tactile, auditory and inertial sensor data, and adapts to the changing environment and task requirements by real-time updating of the environment perception model and feedback adjustment strategy, thereby significantly improving the environment perception ability, decision-making accuracy and collaborative operation ability of the robot integrated with multimodal sensing;

[0045] 2. The present invention integrates RGB-D cameras, pressure distribution arrays, microphone arrays and inertial measurement units to ensure that the robot can collect rich multimodal data in real time, including images, contact forces, sounds and posture information, thereby improving its perception of complex working environments;

[0046] 3. The present invention adopts multiple sensors and data fusion technology. The method shows stronger robustness and adaptability when facing complex or changing environments. Even if a sensor fails or the data is inaccurate, other sensors can still provide useful information, thereby ensuring that the robot can continue to perform the task. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the working steps of the present invention;

[0048] Figure 2 This is a schematic diagram of the hardware structure of the robot body of the present invention;

[0049] Figure 3 The structural schematic diagram is described in detail for the working steps of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Example 1: Please refer to Figure 1 , Figure 2 and Figure 3 , an embodiment of the present invention provides: a multi-modal sensing integrated robot collaborative operation method, comprising the following steps:

[0052] Step S1, data acquisition: by using visual sensors, tactile sensors, auditory sensors and inertial sensors to form a sensor array, multimodal sensor data of the robot's working environment is acquired in real time, including image sequences, contact force distribution matrices, voiceprint spectrograms and robot six-degree-of-freedom posture information;

[0053] Step S2, data preprocessing: preprocess the collected multimodal data, including denoising, filtering, spatiotemporal calibration and fault-tolerant processing, to ensure the quality and consistency of the data. When any sensor fails or is damaged, other sensors can continue to work and provide effective data support to ensure that the robot can continue to perform tasks.

[0054] Step S3, data fusion: using a data fusion algorithm to fuse the pre-processed multimodal data to generate a comprehensive environment perception model, which can more comprehensively reflect the actual situation of the robot's environment, including the location, shape, material and dynamic changes of objects;

[0055] Step S4, feature extraction and recognition: extract key features based on the fused multimodal data, and use machine learning or deep learning algorithms to perform pattern recognition to identify objects, people or sound data in the environment, including the category of objects, the actions of people and the source of sounds;

[0056] Step S5, decision making: according to the recognition results, combined with the robot's task objectives and current state, make collaborative operation decisions, including path planning, action selection and interaction strategies, and judge in real time whether the decision can be completed accurately and the safety of the robot during movement, to ensure the reliability and safety of the operation, and during the movement, detect the path environment to determine whether there are potential dangers in the planned path;

[0057] Step S6, collaborative execution: The robot performs collaborative operations with other robots or humans based on the decision results to perform the corresponding tasks, while updating the environmental perception model in real time, and adjusting the decision and execution strategies in a timely manner based on the feedback adjustment steps to adapt to the changing environment and task requirements;

[0058] Furthermore, the robot collaborative operation method based on multimodal sensing integration integrates visual, tactile, auditory and inertial sensor data to achieve comprehensive and real-time perception of the robot's working environment. First, multimodal sensing data is collected in real time and preprocessed to ensure data quality and consistency. Subsequently, a data fusion algorithm is used to generate a comprehensive environmental perception model that can accurately reflect the actual situation of the environment. Based on the model, the robot can extract key features and perform pattern recognition to accurately identify objects, people or sound data in the environment. According to the recognition results and the robot's task objectives and current status, the robot can make collaborative operation decisions and detect the path environment in real time during movement to ensure the safety and reliability of the operation. Finally, the robot collaborates with other robots or humans, updates the environmental perception model in real time and adjusts the strategy based on feedback to adapt to the changing environment and task requirements, thereby significantly improving the robot's environmental perception ability, decision-making accuracy and collaborative operation ability.

[0059] By adopting multiple sensors and data fusion technology, this method shows greater robustness and adaptability when facing complex or changing environments. Even if a sensor fails or the data is inaccurate, other sensors can still provide useful information, ensuring that the robot can continue to perform the task.

[0060] Example 2: Please refer to Figure 2 and Figure 3 , an embodiment provided by the present invention: in the step S1, during the data collection process, the visual sensor uses an RGB-D camera to collect and obtain an RGB-D image sequence, which is used to capture the RGB-D image sequence of the robot's working environment and provide visual information;

[0061] The tactile sensor uses a pressure distribution array to collect and obtain the contact force distribution matrix, which is used to detect the contact force distribution between the robot and the environment and provide tactile feedback;

[0062] The auditory sensor uses a microphone array to collect and obtain voiceprint spectrograms to capture sounds in the environment and provide auditory information;

[0063] The inertial sensor uses an inertial measurement unit to collect and obtain the robot's six-degree-of-freedom posture information, including position, velocity, and acceleration information;

[0064] Furthermore, by integrating RGB-D cameras, pressure distribution arrays, microphone arrays and inertial measurement units, the robot can collect rich multimodal data in real time, including images, contact forces, sounds and posture information, thereby improving its perception of complex working environments.

[0065] Example 3: Please refer to Figure 3, an embodiment provided by the present invention: in the step S2, in the data preprocessing process, wavelet threshold denoising (threshold coefficient 0.8) and Kalman filtering (process noise covariance 0.01) are used;

[0066] Filtering, used to remove unnecessary fluctuations and jitters;

[0067] Space-time calibration: μs-level time synchronization is achieved through hardware trigger pulses, and the spatial grid calibration method is used to compensate for sensor position deviation;

[0068] Fault-tolerance processing: Design a redundant sensor voting mechanism to ensure that the system availability is ≥ 95% when a single sensor fails;

[0069] Furthermore, through the synergistic effect of wavelet threshold denoising (threshold coefficient 0.8) and Kalman filtering (process noise covariance 0.01), the system effectively filters out high-frequency noise and motion jitter, thereby improving the sensor signal-to-noise ratio by more than 12dB and reducing the pose estimation error to ±1.2cm. The spatiotemporal calibration link uses hardware trigger pulses to achieve μs-level time synchronization, and combines the spatial grid calibration method to compensate for the sensor position deviation, ensuring the spatiotemporal consistency of multimodal data and improving spatial positioning accuracy. In addition, the redundant sensor voting mechanism uses weighted decision-making to maintain system availability ≥95% when a single sensor fails, which is conducive to improving the stability of the task completion rate and providing highly robust and low-latency decision support for robot collaborative operations.

[0070] Example 4: Please refer to Figure 3 , an embodiment provided by the present invention: in the step S3, in the data fusion step, an adaptive weighted fusion algorithm based on information entropy is adopted, and the formula is expressed as:

[0071] ;

[0072] Among them, H i is the information entropy of the i-th sensor, and the calculation formula is:

[0073] ;

[0074] Among them, W i represents the weight of the i-th data, H i represents the information entropy of the i-th sensor, H j represents the information entropy of the jth sensor, represents an exponential function with the natural constant e as the base, in which the information entropy value is converted into a non-negative weight coefficient. The larger the entropy value, the The smaller it is, the more adaptive weight adjustment can be achieved. The summation symbol is used to calculate the sum of all items from j=1 to j=n, indicating the traversal calculation of n sensor data. represents the probability value of the kth feature of the i-th sensor, represents the number of feature dimensions, It is a logarithmic function used to calculate information entropy;

[0075] Furthermore, by adopting an adaptive weighted fusion algorithm based on information entropy, the preprocessed multimodal data are effectively fused to generate a comprehensive environmental perception model, which can more comprehensively reflect the actual situation of the environment and provide accurate data support for the robot's decision making.

[0076] Example 5: Please refer to Figure 3 , an embodiment provided by the present invention: in the step S4, in the feature extraction and recognition step, a multimodal feature space is constructed; including visual features, tactile features, auditory features and inertial features, and an improved YOLOv5 network is used for multimodal target detection, achieving mAP@0.5≥92% on a custom warehouse data set;

[0077] Furthermore, by constructing a multimodal feature space that includes visual, tactile, auditory and inertial features, the system achieves all-round perception of warehouse environment information. The improved YOLOv5 network deeply integrates multimodal data and demonstrates excellent detection performance on a custom warehouse dataset. The mAP@0.5 index reaches more than 92%, a huge improvement over the basic version. The multimodal collaborative mechanism improves target detection accuracy.

[0078] Example 6: Please refer to Figure 2 and Figure 3 ,An embodiment provided by the present invention: In the decision making step, a two-layer decision architecture is designed: Path planning layer: the RRT*-Connect algorithm is used to generate the obstacle avoidance path, and the path cost function is adjusted in real time through tactile feedback;

[0079] Action selection layer: Based on the DQN reinforcement learning framework, multimodal fusion features are used as state input to train a policy network with a grasping success rate of ≥ 98%;

[0080] Furthermore, the robot can make collaborative operation decisions based on the recognition results and task objectives combined with the current status. At the same time, by detecting the path environment, it can determine whether the planned path actions have potential dangers, thereby ensuring the safety and reliability of the operation.

[0081] Example 7: Please refer to Figure 2 and Figure 3, an embodiment provided by the present invention: in the step S6, in the collaborative execution step, the robot communicates with other robots or humans in real time through a wireless communication protocol, shares the environmental perception models collected and processed by sensors and the decision results based on these models, and these shared information allow the robots to perform precise collaborative operations to achieve common task goals;

[0082] Furthermore, robots can communicate with other robots or humans in real time through wireless communication protocols, share environmental perception models and decision-making results, and achieve efficient collaborative operations, which helps to improve the overall work efficiency of the team and reduce the difficulty and complexity of collaborative operations.

[0083] Example 8: Please refer to Figure 3 , an embodiment provided by the present invention: the feedback adjustment step in step S6 is used to adjust the environment perception model, decision-making and execution strategy in real time according to the task execution status and environmental changes during the collaborative execution process to improve the efficiency and accuracy of the collaborative operation;

[0084] Furthermore, during the collaborative execution process, the robot can adjust the environmental perception model, decision-making and execution strategy in real time according to the task execution status and environmental changes. This flexible feedback adjustment mechanism helps to improve the efficiency and accuracy of collaborative operations, enabling the robot to better adapt to the changing environment and task requirements.

[0085] Working principle: The robot collaborative operation method integrated with multimodal sensing realizes comprehensive and real-time perception of the robot's working environment by integrating visual, tactile, auditory and inertial sensor data. First, multimodal sensing data is collected in real time and preprocessed to ensure data quality and consistency. Then, a data fusion algorithm is used to generate a comprehensive environmental perception model, which can accurately reflect the actual situation of the environment. Based on this model, the robot can extract key features and perform pattern recognition to accurately identify objects, personnel or sound data in the environment. According to the recognition results and the robot's task objectives and current status, the robot can make collaborative operation decisions and detect the path environment in real time during movement to ensure the safety and reliability of the operation. Finally, the robot collaborates with other robots or humans, updates the environmental perception model in real time and adjusts the strategy based on feedback to adapt to the changing environment and task requirements, thereby significantly improving the robot's environmental perception ability, decision-making accuracy and collaborative operation ability. By integrating RGB-D cameras, pressure distribution arrays, microphone arrays and inertial measurement units, it is ensured that the robot can collect rich multimodal data in real time, including images, contact forces, sounds and posture information, thereby improving its perception of complex working environments. In the data preprocessing step, wavelet threshold denoising and Kalman filtering technology are used to purify the signal, and data synchronization and calibration are performed to eliminate noise and errors between sensors and ensure the accuracy and consistency of the data. By using the data fusion algorithm of the weighted average method, the preprocessed multimodal data is effectively fused to generate a comprehensive environmental perception model. The model can more comprehensively reflect the actual situation of the environment and provide accurate data support for the robot's decision-making. The robot can make collaborative operation decisions based on the recognition results and task objectives combined with the current state. At the same time, by detecting the path environment, it can be determined whether the planned path action has potential dangers to ensure the safety and reliability of the operation. The robot can communicate with other robots or humans in real time through wireless communication protocols, share environmental perception models and decision results, and achieve efficient collaborative operation, which helps to improve the overall work efficiency of the team and reduce the difficulty and complexity of collaborative operation. During the collaborative execution process, the robot can adjust the environmental perception model, decision-making and execution strategy in real time according to the task execution and environmental changes. This flexible feedback adjustment mechanism helps to improve the efficiency and accuracy of collaborative operation and enable the robot to better adapt to the changing environment and task requirements.

[0086] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be considered exemplary and non-restrictive in all respects, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.

Claims

1. A multi-modal sensing integrated robot collaborative operation method, characterized in that: The following steps are involved: Step S1, data acquisition: by using visual sensors, tactile sensors, auditory sensors and inertial sensors to form a sensor array, multimodal sensor data of the robot's working environment is acquired in real time, including image sequences, contact force distribution matrices, voiceprint spectrograms and robot six-degree-of-freedom posture information; Step S2, data preprocessing: preprocess the collected multimodal data, including denoising, filtering, spatiotemporal calibration and fault-tolerant processing, to ensure the quality and consistency of the data. When any sensor fails or is damaged, other sensors can continue to work and provide effective data support to ensure that the robot can continue to perform tasks. Step S3, data fusion: using a data fusion algorithm to fuse the pre-processed multimodal data to generate a comprehensive environment perception model, which can more comprehensively reflect the actual situation of the robot's environment, including the location, shape, material and dynamic changes of objects; Step S4, feature extraction and recognition: extract key features based on the fused multimodal data, and use machine learning or deep learning algorithms to perform pattern recognition to identify objects, people or sound data in the environment, including the category of objects, the actions of people and the source of sounds; Step S5, decision making: according to the recognition results, combined with the robot's task objectives and current state, make collaborative operation decisions, including path planning, action selection and interaction strategies, and judge in real time whether the decision can be completed accurately and the safety of the robot during movement, to ensure the reliability and safety of the operation, and during the movement, detect the path environment to determine whether there are potential dangers in the planned path; Step S6, collaborative execution: The robot collaborates with other robots or humans to perform corresponding tasks based on the decision results, updates the environmental perception model in real time, and adjusts the decision and execution strategies in time according to the feedback adjustment steps to adapt to the ever-changing environment and task requirements.

2. The multi-modal sensing integrated robot collaborative operation method according to claim 1, characterized in that: In step S1, during the data collection process, the visual sensor uses an RGB-D camera to collect and obtain an RGB-D image sequence, which is used to capture the RGB-D image sequence of the robot's working environment and provide visual information; The tactile sensor uses a pressure distribution array to collect and obtain the contact force distribution matrix, which is used to detect the contact force distribution between the robot and the environment and provide tactile feedback; The auditory sensor uses a microphone array to collect and obtain voiceprint spectrograms to capture sounds in the environment and provide auditory information; The inertial sensor uses an inertial measurement unit to collect and obtain the robot's six-degree-of-freedom posture information, including position, velocity, and acceleration information.

3. The multi-modal sensing integrated robot collaborative operation method according to claim 1, characterized in that: In step S2, in the data preprocessing process, wavelet threshold denoising (threshold coefficient 0.8) and Kalman filtering (process noise covariance 0.01) are used; Filtering, used to remove unnecessary fluctuations and jitters; Space-time calibration: μs-level time synchronization is achieved through hardware trigger pulses, and the spatial grid calibration method is used to compensate for sensor position deviation; Fault-tolerance processing: Design a redundant sensor voting mechanism to ensure that system availability is ≥ 95% when a single sensor fails.

4. The multi-modal sensing integrated robot collaborative operation method according to claim 1, characterized in that: In the step S3, in the data fusion step, an adaptive weighted fusion algorithm based on information entropy is adopted, and the formula is expressed as follows: ; Among them, H i is the information entropy of the i-th sensor, and the calculation formula is: ; Among them, W i represents the weight of the i-th data, H i represents the information entropy of the i-th sensor, H j represents the information entropy of the jth sensor, represents an exponential function with the natural constant e as the base, in which the information entropy value is converted into a non-negative weight coefficient. The larger the entropy value, the The smaller it is, the more adaptive weight adjustment can be achieved. The summation symbol is used to calculate the sum of all items from j=1 to j=n, indicating the traversal calculation of n sensor data. represents the probability value of the kth feature of the i-th sensor, represents the number of feature dimensions, It is a logarithmic function used to calculate information entropy.

5. The multi-modal sensing integrated robot collaborative operation method according to claim 1, characterized in that: In the step S4, in the feature extraction and recognition step, a multimodal feature space is constructed, including visual features, tactile features, auditory features and inertial features, and an improved YOLOv5 network is used for multimodal target detection, achieving mAP@0.5≥92% on a custom warehouse dataset.

6. The multi-modal sensing integrated robot collaborative operation method according to claim 1, characterized in that: In step S5, in the decision-making step, a two-layer decision-making architecture is designed: Path planning layer: The RRT*-Connect algorithm is used to generate obstacle avoidance paths, and the path cost function is adjusted in real time through tactile feedback; Action selection layer: Based on the DQN reinforcement learning framework, multimodal fusion features are used as state input to train a policy network with a grasping success rate of ≥ 98%.

7. The multi-modal sensing integrated robot collaborative operation method according to claim 1, characterized in that: In step S6, in the collaborative execution step, the robot communicates with other robots or humans in real time through wireless communication protocols, shares the environmental perception models collected and processed by sensors and the decision results made based on these models, and these shared information allow the robots to perform precise collaborative operations to achieve common task goals.

8. The multi-modal sensing integrated robot collaborative operation method according to claim 1, characterized in that: The feedback adjustment step in step S6 is used to adjust the environmental perception model, decision-making and execution strategy in real time according to the task execution status and environmental changes during the collaborative execution process to improve the efficiency and accuracy of the collaborative operation.

Images