Robot motion planning method and device based on multi-modal information fusion

Abstract

The invention relates to the field of robot motion, in particular to a multi-modal information fusion robot motion planning method, which comprises the following steps: information acquisition: acquiring multi-modal data of a robot through various sensors; preprocessing: carrying out normalization, denoising and synchronization processing on the multi-modal data; fusion: coding and fusing the preprocessed data by adopting a feature level fusion method, and generating an environment model based on an information fusion algorithm; motion planning: combining an environment modeling and path planning algorithm, and generating a motion path according to the fused multi-modal information; and feedback adjustment: dynamically adjusting the motion track by adopting a feedback control mechanism based on the environment change sensed in real time. The invention provides a multi-modal information fused robot motion planning method and equipment, which are used for improving the perception capability of a robot to a complex environment, enhancing motion planning and precision and improving task execution efficiency and safety by fusing multi-modal information.

Description

Technical Field

[0001] The present invention relates to the field of robot motion, and in particular to a robot motion planning method and device for multi-modal information fusion. Background Art

[0002] Current robot motion planning relies on single-mode perception data (such as vision or lidar), which results in insufficient environmental perception and decision-making capabilities in complex scenarios. For example, single vision is susceptible to low-light conditions, and lidar has limited perception capabilities for transparent or smooth objects.

[0003] Therefore, how to utilize the fusion of multimodal perception information (such as vision, touch, lidar, and voice) to improve the accuracy of robot motion planning has become the key to technological development. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a robot motion planning method and device with multimodal information fusion, which is used to improve the robot's perception ability of complex environments, enhance motion planning and accuracy, and improve task execution efficiency and safety by fusing multimodal information.

[0005] The specific technical solutions are as follows:

[0006] The present application provides a robot motion planning method for multimodal information fusion, comprising the steps of:

[0007] Step S1: Acquisition of multimodal information:

[0008] Environmental information is obtained in real time through a variety of sensors (such as cameras, lidar, tactile sensors, microphones, etc.), including but not limited to images, depth data, contact torque, voice commands, etc.

[0009] Step S2: Data preprocessing:

[0010] The data of each modality are normalized, denoised and synchronized.

[0011] The data formats of different modalities are unified by using timestamp and coordinate conversion methods.

[0012] Step S3: Multimodal information fusion:

[0013] Low-level fusion: Based on methods such as convolutional neural network (CNN) and recurrent neural network (RNN), images, lidar point clouds and tactile information are encoded into feature vectors.

[0014] High-level fusion: Based on graph neural networks (GNNs) or attention mechanisms, low-level features are uniformly mapped to a shared semantic space to achieve collaborative understanding among multiple modalities.

[0015] Step S4: Motion planning

[0016] Combine environmental modeling with path planning algorithms (such as A* algorithm, RRT algorithm or deep reinforcement learning) to generate motion paths based on the fused multimodal information.

[0017] Think dynamic obstacle avoidance, voice command prioritization, and instant responsiveness with haptic feedback.

[0018] Step S5: Feedback adjustment mechanism

[0019] Based on real-time perceived environmental changes, a feedback control mechanism is used to dynamically adjust the motion trajectory.

[0020] Continuously optimize the planning model through reinforcement learning to improve decision-making efficiency.

[0021] As a further elaboration of the scheme, the present application also provides a multi-modal information fusion robot motion planning device, comprising:

[0022] Multimodal sensor module: includes camera, lidar, microphone and tactile sensor.

[0023] Data processing module: High-performance embedded processors (such as NVIDIA Jetson or FPGA) are used for data fusion calculations.

[0024] Control module: Robot motion controller, used to execute the path instructions of motion planning.

[0025] Communication module: supports real-time interaction with external devices, such as cloud computing or remote monitoring.

[0026] Technical effects and advantages of the present invention:

[0027] 1. Improve the robot's perception of complex environments, enhance motion planning and accuracy by integrating multimodal information, and reduce the blind spots of single-modal perception.

[0028] 2. It can realize adaptive path planning in dynamic environments and improve the efficiency and safety of task execution.

[0029] 3. Strong scalability, suitable for a variety of robot application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flowchart of a robot motion planning method for multimodal information fusion provided in an embodiment of the present application;

[0031] Figure 2 It is a schematic diagram of the composition of a multimodal information fusion robot motion planning device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention are provided for the purpose of illustration and description, and are not intended to be exhaustive or to limit the present invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention, and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

[0033] Example 1

[0034] See also Figure 1 In this embodiment, a robot motion planning method for multimodal information fusion is provided, comprising the following steps:

[0035] Step S1, information acquisition: acquiring multimodal data of the robot through a variety of sensors.

[0036] Among them, multimodal information includes but is not limited to: images, depth data, contact torque, and voice command data obtained by using cameras, lidar, tactile sensors, and microphone devices respectively.

[0037] Step S2, preprocessing: normalize, denoise and synchronize the multimodal data.

[0038] Wherein, in step S2, the timestamp and coordinate conversion method are used to unify the data formats of different modes, specifically including:

[0039] S2.1: Align multimodal data based on timestamps and use interpolation methods to fill missing values;

[0040] S2.2: Use a calibration algorithm (such as chessboard calibration method) to calibrate the spatial position of the camera and lidar, and unify all data into the same coordinate system;

[0041] S2.3: Use filtering and edge detection methods to extract features from images, and use VoxelNet to perform sparse and block processing on point cloud data to improve computing efficiency.

[0042] Step S3, fusion: use feature-level fusion method to encode and fuse the preprocessed data.

[0043] The feature level fusion method in step S3 includes low-level fusion and high-level fusion.

[0044] The low-level fusion described in S3.1 includes: encoding images, lidar point clouds and tactile information into feature vectors based on methods of convolutional neural networks and recurrent neural networks; each modality needs to extract features independently and transform them into a shared feature space, including:

[0045] 1) Visual modality:

[0046] The convolutional neural network (CNN) is used to extract the edge, texture and semantic features of the object. The network can use the ResNet or EfficientNet architecture.

[0047] 2) LiDAR mode:

[0048] Use a 3D point cloud network (such as PointNet++) to extract the shape and spatial position features of obstacles and classify the obstacles.

[0049] 3) Tactile modality:

[0050] The tactile time series data is processed based on the LSTM network to extract dynamic features such as contact force and vibration to determine the grasping or contact state.

[0051] 4) Voice mode:

[0052] Use the Transformer model to parse user voice commands and extract task-related semantic information.

[0053] S3.2 High-level fusion includes: based on graph neural networks or attention mechanisms, low-level features are uniformly mapped to a shared semantic space to achieve collaborative understanding among multiple modalities. Specifically, the low-level features of each modality are further integrated to build a unified environmental understanding model, including:

[0054] S3.2.1 Graph Neural Network (GNN) Fusion:

[0055] A graph structure is constructed, and the feature vector of each mode is regarded as a node of the graph.

[0056] S3.2.2 uses the attention mechanism (Graph Attention Network, GAT) to calculate the correlation between nodes and generate a fused feature representation.

[0057]

[0058] Among them, α ij is the relevance weight of nodes i and j, calculated by the attention mechanism.

[0059] S3.2.3 Adaptive weight allocation:

[0060] The modal weights are dynamically allocated according to the environment. The formula is as follows:

[0061]

[0062] Among them, w i is the weight of modality i, which is determined by the confidence of the modality and the environmental conditions. The weights of each modality are dynamically adjusted based on the scene requirements and sensor reliability. For example, the weight of lidar is increased in low-light conditions, while visual information is prioritized in spacious environments.

[0063] Step S4, motion planning: combining environment modeling with path planning algorithm, generating a motion path according to the fused multimodal information.

[0064] The path planning algorithm in step S4 adopts A* algorithm, RRT algorithm or deep reinforcement learning, while considering dynamic obstacle avoidance, voice command priority and instant response of tactile feedback.

[0065] The fused features are used to generate motion paths, taking into account changes in dynamic environments:

[0066] Among them, adaptive planning based on reinforcement learning, taking the Deep Reinforcement Learning (DRL) algorithm as an example, trains robots to adapt to obstacles and dynamic changes in complex environments, including:

[0067] State: The environment information represented by the fused features.

[0068] Action: Robot movement instructions (such as turning, accelerating).

[0069] Reward: Design the reward function by considering factors such as the shortest path and obstacle avoidance success rate.

[0070] Multi-objective optimization: Through the non-dominated sorting genetic algorithm (NSGA-II), multiple objectives (such as safety, energy consumption, and time) in path planning are optimized.

[0071] Step S5, feedback adjustment: Based on real-time perceived environmental changes, the motion trajectory is dynamically adjusted using a feedback control mechanism, and the planning model is continuously optimized through reinforcement learning to improve decision-making efficiency.

[0072] Example 2

[0073] See also Figure 2 As shown, this embodiment discloses a robot motion planning device for multi-modal information fusion, including:

[0074] Multimodal sensor module: includes camera, lidar, microphone and tactile sensor;

[0075] Data processing module: The processor is used for data fusion calculation;

[0076] Control module: robot motion controller, used to execute the path instructions of motion planning;

[0077] Communication module: supports real-time interaction with external devices, using cloud computing or remote monitoring.

[0078] The steps of the robot motion planning method for multimodal information fusion described in Example 1 are executed by the above-mentioned device.

[0079] Example 3

[0080] This embodiment provides a usage scenario of a home service robot.

[0081] It can use vision to identify obstacles, voice to receive user commands, and combine tactile information to achieve precise grasping and path planning.

[0082] Example 4

[0083] This embodiment provides a usage scenario of an industrial handling robot.

[0084] By combining lidar and visual information, dynamic obstacle avoidance and efficient path planning in complex factory environments can be achieved.

[0085] Example 5

[0086] This embodiment provides a usage scenario of a rescue robot.

[0087] Multimodal fusion is used to achieve the effect of on-site search and rescue. In addition to the images, depth data, contact torque, and voice command data obtained by cameras, lidars, tactile sensors, and microphone devices, the multimodal fusion also includes thermal imaging recognition by infrared sensors, searching for and contacting survivors in complex environments, and planning the optimal rescue path.

[0088] Example 6

[0089] This embodiment provides a usage scenario in a flexible manufacturing robot.

[0090] Utilize multimodal information to perceive the production line and adjust the robot's path in real time to cope with changes in the position of workpieces on the assembly line.

[0091] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field and related fields without creative work should fall within the scope of protection of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention are implemented according to the conventional means in the field unless otherwise specified and limited.

Claims

1. A robot motion planning method based on multimodal information fusion, characterized in that: The steps include: Step S1, information acquisition: acquiring multimodal data of the robot through a variety of sensors; Step S2, preprocessing: normalizing, denoising and synchronizing the multimodal data; Step S3, fusion: using feature-level fusion method to encode and fuse the preprocessed data, and generating an environment model based on the information fusion algorithm; Step S4, motion planning: combining environment modeling with path planning algorithm to generate a motion path according to the fused multimodal information; Step S5, feedback adjustment: Based on real-time perceived environmental changes, the motion trajectory is dynamically adjusted using a feedback control mechanism, and the planning model is continuously optimized through reinforcement learning to improve decision-making efficiency.

2. The robot motion planning method of multimodal information fusion according to claim 1, characterized in that: The multimodal information in step S1 includes but is not limited to: images, depth data, contact torque, and voice command data obtained by using a camera, a laser radar, a tactile sensor, and a microphone device respectively.

3. The robot motion planning method of multimodal information fusion according to claim 2, characterized in that: In step S2, the timestamp and coordinate conversion method are used to unify the data formats of different modes, specifically including: S2.1: Align multimodal data based on timestamps and use interpolation methods to fill missing values; S2.2: Use the calibration algorithm to calibrate the spatial position of the camera and lidar, and unify all data into the same coordinate system; S2.3: Use filtering and edge detection methods to extract features from images, and use VoxelNet to perform sparse and block processing on point cloud data.

4. The robot motion planning method of multimodal information fusion according to claim 3, characterized in that: The feature-level fusion method in step S3 includes low-level fusion and high-level fusion, wherein the low-level fusion includes: encoding the image, the lidar point cloud and the tactile information into a feature vector based on a convolutional neural network and a recurrent neural network; The high-level fusion includes: based on graph neural networks or attention mechanisms, low-level features are uniformly mapped to a shared semantic space to achieve collaborative understanding among multiple modalities.

5. The robot motion planning method of multimodal information fusion according to claim 4, characterized in that: The path planning algorithm in step S4 adopts A* algorithm, RRT algorithm or deep reinforcement learning, while considering dynamic obstacle avoidance, voice command priority and instant response of tactile feedback.

6. A robot motion planning device with multimodal information fusion, characterized in that: include: Multimodal sensor module: includes camera, lidar, microphone and tactile sensor; Data processing module: The processor is used for data fusion calculation; Control module: robot motion controller, used to execute the path instructions of motion planning; Communication module: supports real-time interaction with external devices, using cloud computing or remote monitoring; The above-mentioned device is used to execute the steps of the robot motion planning method for multimodal information fusion as described in any one of claims 1 to 5.

Images