Robot obstacle avoidance and navigation method based on multimodal fusion and visual language model

Abstract

This invention provides a robot obstacle avoidance and navigation method based on multimodal fusion and a visual language model. The method involves acquiring real-time data from different modalities using multiple sensors, performing time synchronization and normalization processing; extracting features from the different modal data and fusing them through an intermediate layer; inputting the fused multimodal data into a visual language model, and using the model's semantic segmentation and object detection results to generate a semantic map of the environment. Action strategies are generated by combining natural language instructions and visual analysis results, and the generated action strategies are converted into control signals executable by the robot to achieve closed-loop control. By combining visual language models with multimodal sensor fusion technology, this invention aims to enhance the perception, decision-making, and real-time response capabilities of mobile robots in complex and dynamic environments.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile robots, and in particular to a robot obstacle avoidance and navigation method based on multimodal fusion and a visual language model. Background Art

[0002] With the rapid development of artificial intelligence and robotics, mobile robots, thanks to their high degree of automation and flexibility, are increasingly being used in a wide range of fields, including industry, services, and healthcare. In industrial production, mobile robots can efficiently complete tasks such as material handling and assembly, significantly improving production efficiency and reducing labor costs. In the service sector, they can provide services such as guided tours and cleaning, bringing convenience to people's lives. In healthcare scenarios, mobile robots can assist medical staff with tasks such as drug delivery and patient transport, reducing their workload.

[0003] However, the traditional methods currently relied upon by mobile robots for obstacle avoidance and navigation have many limitations, which severely restrict their application effectiveness in complex dynamic environments. The specific manifestations are as follows:

[0004] Traditional mobile robots typically use a single sensor for environmental perception, such as lidar or cameras. Lidar acquires distance information from surrounding objects by emitting a laser beam and measuring the time it takes for the reflected light to arrive. While capable of accurately measuring distance, it lacks the ability to perceive semantic information such as an object's color and texture. Cameras excel at capturing visual information, but their image quality is severely affected in complex lighting conditions such as low light and strong interference, leading to distorted environmental perception. In complex and dynamic environments, objects of various shapes, materials, and motions often exist. A single sensor struggles to fully and accurately perceive this information, failing to provide a reliable basis for robot environmental cognition and increasing the risk of collisions with obstacles.

[0005] Traditional obstacle avoidance and navigation algorithms primarily make decisions based on simple rules or preset map information, lacking a deep understanding of semantic information. When faced with natural language instructions, traditional algorithms struggle to effectively correlate the semantic content of the instructions with actual environmental information, failing to accurately understand the true intent of the instructions. For example, when an operator gives a natural language instruction such as "Go to the meeting room, avoiding the moving cleaning truck," traditional algorithms may not be able to accurately identify the location of the "meeting room" or the movement of the "cleaning truck," resulting in a slow and error-prone decision-making process and an inability to quickly and accurately plan a suitable path.

[0006] Traditional methods often require extensive computing resources for environmental perception and decision-making tasks. Due to the limited computing power of robots themselves, many complex computational tasks must rely on cloud servers. However, the time required to transmit data between the robot and the cloud results in high system latency. In dynamic environments, where environmental information changes rapidly, this high latency prevents robots from responding to environmental changes in a timely manner, making it difficult to meet the requirements of real-time obstacle avoidance and navigation, severely impacting their operational efficiency and safety.

[0007] In recent years, multimodal visual language models (such as Qwen) have demonstrated powerful capabilities in image understanding, semantic reasoning, and cross-modal interaction. Multimodal visual language models can simultaneously process information from multiple modalities, such as images and text. By learning from large amounts of data, they acquire rich semantic knowledge and reasoning capabilities. They can not only accurately understand information such as objects and scenes in images, but also make reasonable inferences and decisions based on textual instructions.

[0008] However, integrating these advanced model capabilities with multimodal sensor fusion technology and successfully applying them to obstacle avoidance and navigation tasks for mobile robots remains a critical challenge in robotics. By fusing multimodal visual language models with multimodal sensors, it is hoped that the advantages of both approaches will be fully leveraged, effectively overcoming the limitations of traditional methods and providing a more efficient and accurate solution for obstacle avoidance and navigation for mobile robots in complex and dynamic environments. Summary of the Invention

[0009] In response to the deficiencies in the existing technologies in environmental perception, decision-making and real-time control, the purpose of the present invention is to provide a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model, combining the multimodal visual language model with multimodal sensor fusion technology, aiming to enhance the perception, decision-making and real-time response capabilities of mobile robots in complex dynamic environments.

[0010] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0011] A robot obstacle avoidance and navigation method based on multimodal fusion and visual language model includes the following steps:

[0012] Collect different modal data in real time through multiple sensors, and perform time synchronization and normalization processing;

[0013] Extract features from different modal data and fuse them through an intermediate layer, where the intermediate layer is a graph neural network that fuses image features, point cloud features, and IMU features to generate a unified perception output;

[0014] The fused multimodal data is input into the visual language model, and the semantic segmentation and target detection results of the model are used to generate a semantic map of the environment. The visual input is encoded using a visual encoder to generate a visual feature vector, and the language input is encoded using a language encoder to generate a language feature vector. The interaction between visual features and language features is achieved through the cross-attention mechanism to generate cross-modal features. f cross , expressed as the following formula:

[0015]

[0016] in, α i,j Represents the attention weight, which is used to measure the language feature vector f lang The i elements and visual feature vectors f vis,j The j The degree of correlation between elements, f vis,j is the first j elements, W Q 、W K 、 W V is the learnable weight matrix, d is the scaling factor, T Represents the transpose of a matrix;

[0017] Use multi-layer Transformer modules to process cross-modal features and generate visual analysis results;

[0018] Combine natural language instructions and visual analysis results to generate action strategies;

[0019] The generated action strategy is converted into a control signal that can be executed by the robot to achieve closed-loop control.

[0020] According to a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model provided by the present invention, when performing time synchronization processing, the image acquisition device, lidar and IMU device are coarsely time-aligned using PPS pulse signals or shared clock sources;

[0021] For modal data that are not strictly synchronized, a timestamp-based interpolation or resampling method is used to align discrete time points using a linear interpolation formula, which is expressed as:

[0022]

[0023] in, x(t) For time t The interpolated data at x k and x k+1 adjacent timestamps t k and t k+1 The original data at

[0024] For modes with nonlinear time drift, the Kalman filter is used to estimate the time offset Δt, and the timestamp is dynamically adjusted through the state update equation, which is expressed as:

[0025]

[0026] in, K k is the Kalman gain, t ref is the reference time, t imu,k IMU timestamp.

[0027] According to a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model provided by the present invention, for image features, a pre-trained CNN model is used to extract features from images captured by an image acquisition device to obtain high-level semantic feature vectors;

[0028] Perform global average pooling on the output of the intermediate layer or fully connected layer of the CNN model to compress the three-dimensional feature map C×H×W into a one-dimensional feature vector C×1. The calculation formula is:

[0029]

[0030] in, F c,i,j The cth channel of the feature map is at position (i,j) The activation value at f img is the compressed image feature vector.

[0031] According to a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model provided by the present invention, for point cloud features, a PointNet or PointNet++ network is used to extract geometric features from point cloud data collected by a lidar, specifically including:

[0032] Represent the original point cloud data as an N×3 matrix, where N is the number of point clouds and each row contains three-dimensional coordinates (x, y, z);

[0033] The multi-layer perceptron MLP is used to embed features of each point to generate a high-dimensional feature vector, and the feature space is aligned and transformed through the T-Net network. The calculation formula of the transformation matrix T is:

[0034]

[0035] in, P is the original point cloud matrix, P' is the transformed point cloud matrix, MLP θ is a multilayer perceptron with parameter θ;

[0036] Use maximum pooling to aggregate the features of all points to generate a global point cloud feature vector f lidar , expressed as:

[0037]

[0038] in, f n is the eigenvector of the nth point, R D is the feature dimension.

[0039] According to a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model provided by the present invention, for IMU features, the accelerometer data a( t )=[ a x ( t ), a y ( t ), a z ( t )] T and gyroscope data oh ( t )=[ oh x ( t ), oh y ( t ), oh z ( t )] T , perform the following processing:

[0040] Use complementary filter or Kalman filter to denoise the raw data and estimate attitude angles, including pitch angle i , roll angle ϕ , yaw angle ψ , the update formula of its complementary filter is:

[0041]

[0042] in, i est is the estimated value, which represents the estimation of a parameter or state at the current moment. α is the learning rate or weighting factor that controls the balance between new information and old estimates, oh y ( t ) is the angular velocity, Δ t is the sampling time interval, a x ( t ) is the acceleration, ||a( t )|| is the acceleration modulus;

[0043] Time-integrate the gyroscope data to generate the rotation quaternion q(t) = [q0(t), q1(t), q2(t), q3(t)] T , its quaternion differential equation is:

[0044]

[0045] Where ⊗ is the quaternion multiplication operator.

[0046] The attitude angle, rotation quaternion and acceleration modulus||a( t )||Combined into IMU features f imu :

[0047]

[0048] According to the present invention, a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model is provided, which comprises:

[0049] Define the graph structure and treat the features of each modality as a node in the graph, where image features correspond to node A, point cloud features correspond to node B, and IMU features correspond to node C. The edges between nodes represent the correlation between modalities, and the edge weights are determined by the correlation between modalities.

[0050] Initialize node features. The initial features of each node are generated by the feature extractor of the corresponding modality. The features of node A are extracted by the CNN model, the features of node B are extracted by the PointNet or PointNet++ network, and the features of node C are generated by the complementary filter or Kalman filter.

[0051] The node features are updated through the message passing mechanism, which is expressed as the following formula:

[0052]

[0053] in, It is l Nodes in the layer i Features, is a node i The set of neighbor nodes of W and is a learnable weight matrix, s is the activation function;

[0054] After multiple rounds of message passing, the global pooling operation is used to aggregate the features of all nodes into a global feature vector, which is the output of multimodal fusion.

[0055] According to the robot obstacle avoidance and navigation method based on multimodal fusion and visual language model provided by the present invention, the image features are f img , point cloud features f lidar and IMU characteristics f imu After fusion, the fused multimodal features are generated f fused :

[0056] The fused multimodal features f fused As visual input, the user's natural language instructions U As language input, construct the input pair of the visual language model ( f fused , U );

[0057] Use visual encoder to process visual input f fused Encode and generate visual feature vector f vis , expressed as the following formula:

[0058] f vis =Vis enc ( f fused ; i vis )

[0059] in, i vis are the visual encoder parameters;

[0060] The target detection algorithm is used to locate the target frame and classify the target to generate the target detection result; combined with the visual feature vector fvis And the target detection results are used to generate a semantic map of the environment through spatial relationship modeling.

[0061] According to a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model provided by the present invention, a language encoder is used to encode the language input U to generate a language feature vector f lang , expressed as the following formula:

[0062] f lang =Lang enc ( U ; i lang )

[0063] in, i lang The language encoder parameters.

[0064] According to the robot obstacle avoidance and navigation method based on multimodal fusion and visual language model provided by the present invention, a multi-layer Transformer module is used to transform cross-modal features. f cross Perform progressive enhancement and output semantic maps M sem , target location T The visual analysis results are expressed as the following formula:

[0065] f enhanced =Transformer N dec ( f cross ; i cross )

[0066] M sem =Decoder map ( f enhanced ; i map )

[0067] T =Detector( f enhanced ; i det )

[0068] A=Policy( f enhanced ; i policy )

[0069] in, f enhanced For the enhanced cross-modal feature vector, Transformer N dec To represent a module consisting of N layers of Transformer decoders, Decoder map is the semantic map decoder, Detector is the target detector, Policy is the policy generation module, i cross 、 i map 、 i det 、 i policy are the parameters of the cross-modal Transformer decoder, semantic map decoder, object detector, and policy generation module respectively;

[0070] Based on semantic map M sem, semantically annotate different areas in the environment; according to the target location T , mark specific targets in the semantic map; combined with action strategy A, update the semantic map in real time to reflect dynamic changes in the environment.

[0071] According to a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model provided by the present invention, the action strategy A includes:

[0072] Inferring target locations and generating waypoint sequences from semantic maps in navigation tasks: Based on cross-modal features f cross , using A* algorithm or Dijkstra algorithm in semantic map M Search the optimal path in sem and generate a path point sequence M. The generated path point sequence M is expressed as {( x 1, y 1),( x 2, y 2),…,( xn , in )},in( xi , yes ) is the i The coordinates of the path points;

[0073] The obstacle avoidance rule formed by using the dynamic window method to plan the path in the obstacle avoidance task is: when a dynamic obstacle is detected, the path is adjusted through the local path replanning algorithm to generate a new path point sequence M′.

[0074] Thus, the present invention proposes a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model. Compared with traditional robot obstacle avoidance and navigation technologies, it has significant and multi-faceted beneficial effects, providing strong support for the efficient operation of robots in various complex scenarios. The specific performance is as follows:

[0075] This invention fuses camera images, lidar point clouds, and IMU data, leveraging the strengths of each sensor. Camera images provide rich visual information, enabling the robot to identify features such as object color, shape, and texture; lidar point clouds accurately measure distance and position, building a detailed three-dimensional environmental model; and IMU data provides real-time information on the robot's motion, including acceleration and angular velocity. By integrating this multimodal data, the robot can perceive its environment from multiple dimensions, building a more comprehensive and accurate understanding of its surroundings and effectively addressing various challenges in complex and dynamic environments. For example, in a home environment, the robot can accurately identify various objects, such as furniture, appliances, and pets, as well as their location and motion, providing a solid data foundation for obstacle avoidance and navigation.

[0076] The method of the present invention can perceive the movement trajectory and speed of dynamic obstacles in real time. The camera image can capture the changes in the obstacle's appearance, the lidar point cloud can accurately measure its position changes, and the IMU data can assist in determining the movement trend of the obstacle. By comprehensively analyzing this multimodal data, the robot can predict the movement path of dynamic obstacles in advance and adjust its own movement strategy in time to effectively avoid collisions and ensure operational safety. For example, in a logistics warehouse, when a worker or forklift suddenly appears in the robot's path, the robot can quickly sense and take evasive action.

[0077] The present invention combines natural language instructions and visual analysis results to generate action strategies. The visual language model conducts in-depth analysis of camera images to identify visual information such as objects, scenes, and signs in the environment, while accurately parsing the intentions, goals, and constraints in natural language instructions. By integrating natural language and visual information, robots can better adapt to different scenarios. Based on the semantic map, the visual analysis results can update environmental information in real time, and natural language instructions can adjust task objectives according to actual conditions. For example, in a shopping mall environment, the robot's original task is to go to a certain store, but if the store location changes or new obstacles are added along the way, the robot can discover the changes through visual analysis, and then combine the user's possible natural language correction instructions to quickly adjust the action strategy and re-plan the path to ensure the smooth completion of the task.

[0078] In obstacle avoidance tasks, the present invention uses a dynamic window method to plan paths and formulate obstacle avoidance rules. Based on the robot's kinematic constraints and current environmental information, the dynamic window method searches for feasible speed combinations within the robot's velocity space, generating a smooth and safe obstacle avoidance path. Combined with environmental perception data provided by multimodal sensor fusion, the robot can perceive the position, speed, and motion trends of surrounding obstacles in real time, dynamically adjust the window size and search range, and quickly plan the optimal obstacle avoidance path.

[0079] In navigation tasks, the present invention infers the target location based on the semantic map and generates a sequence of waypoints. The robot, through its understanding of visual information through a visual language model, can accurately identify landmark objects and areas in the semantic map, thereby precisely inferring the target location. Based on the target location, the robot can generate a reasonable sequence of waypoints to guide itself to its destination. For example, in a hospital environment, the robot can determine the location of different departments based on the semantic map and, combined with the patient's natural language instructions, plan the optimal path from its current location to the target department, improving navigation accuracy and efficiency.

[0080] The method of the present invention is highly versatile and adaptable, and can be widely applied in various scenarios, including household services, logistics and transportation, and industrial production. In household services, robots can perform tasks such as cleaning, delivering items, and providing companionship, bringing convenience to family life. In logistics and transportation, robots can efficiently move goods in complex environments such as warehouses and workshops, improving the efficiency and automation of logistics and transportation. In industrial production, robots can assist workers with tasks such as material handling and assembly, reducing their labor intensity. Through application in various scenarios, the method of the present invention can fully utilize the role of robots, creating greater value for businesses and society.

[0081] In summary, the robot obstacle avoidance and navigation method based on multimodal fusion and visual language model of the present invention significantly improves the robot's perception, decision-making and execution capabilities in complex environments through multi-dimensional perception fusion, natural language interaction support and efficient real-time control, providing strong support for the widespread application and intelligent development of robots, and has broad market prospects and important social significance.

[0082] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 It is a flow chart of an embodiment of a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model of the present invention.

[0084] Figure 2This is a flowchart of time synchronization processing in an embodiment of a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model of the present invention.

[0085] Figure 3 This is a flowchart for extracting image features in an embodiment of a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model of the present invention.

[0086] Figure 4 This is a flowchart for extracting point cloud features in an embodiment of a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model of the present invention.

[0087] Figure 5 This is a flowchart for extracting IMU features in an embodiment of a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model of the present invention.

[0088] Figure 6 This is a flow chart of global feature vector output of multimodal fusion in an embodiment of a robot obstacle avoidance and navigation method based on multimodal fusion and a visual language model of the present invention.

[0089] Figure 7 This is a flowchart for generating a semantic map in an embodiment of a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model of the present invention.

[0090] Figure 8 This is a flowchart for generating visual analysis results in an embodiment of a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model of the present invention.

[0091] Figure 9 It is a system principle diagram of an embodiment of a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model of the present invention. DETAILED DESCRIPTION

[0092] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0093] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0094] See also Figure 1 This embodiment provides a robot obstacle avoidance and navigation method based on multimodal fusion and visual language model, which includes the following steps:

[0095] Step S1, collecting different modal data in real time through multiple sensors, and performing time synchronization processing and normalization processing;

[0096] Step S2: extract features of different modal data and fuse them through the intermediate layer, where the intermediate layer is a graph neural network, which is used to fuse image features, point cloud features, and IMU features to generate a unified perception output;

[0097] Step S3: Input the fused multimodal data into the visual language model and use the model's semantic segmentation and object detection results to identify objects in the image (such as tables and chairs) and their spatial relationships, generate a semantic map of the environment, and mark traversable areas and obstacles. The visual input is encoded using a visual encoder to generate a visual feature vector, and the language input is encoded using a language encoder to generate a language feature vector. A cross-attention mechanism is used to enable interaction between visual and language features to generate cross-modal features. These cross-modal features are then processed using a multi-layer Transformer module to generate visual analysis results.

[0098] Step S4 combines natural language commands and visual analysis results to generate action strategies. This includes using the dynamic windowing algorithm to plan paths and form avoidance rules in obstacle avoidance tasks, and inferring the target location and generating a waypoint sequence based on the semantic map in navigation tasks. For example, if a user issues a command via voice or text: "Take me to the living room," the visual language model, combined with the semantic map, infers the living room's location, generates a waypoint sequence, and uses the dynamic windowing algorithm (DWA) to optimize local obstacle avoidance performance.

[0099] Step S5: Convert the generated action strategy into a control signal that the robot can execute, driving the robot to move along the planned path, achieving closed-loop control. New sensor data is continuously collected to update the path plan in real time.

[0100] In the above step S1, an image acquisition device (such as an RGB camera) is used to capture an environmental image with a resolution of 1920×1080; a lidar is used to collect depth data with a range of 0 to 10 meters; and an inertial measurement unit (IMU) is used to collect acceleration and angular velocity data. Among them, environmental data is collected in real time through multiple sensors such as cameras, lidars, and IMUs (inertial measurement units). The environmental data includes visual data (images), depth data (lidar point clouds), and motion state data (IMU acceleration and angular velocity).

[0101] In the above step S1, the collected multimodal data is time synchronized and format converted to ensure the temporal and spatial consistency of data from different sensors, which is specifically achieved through timestamp alignment and normalization.

[0102] Among them, such as Figure 2 As shown in the figure, when performing time synchronization processing, the image acquisition device, lidar and IMU device are coarsely aligned using PPS pulse signals or shared clock sources;

[0103] For modal data that are not strictly synchronized, a timestamp-based interpolation or resampling method is used to align discrete time points using a linear interpolation formula, which is expressed as:

[0104]

[0105] in, x(t) For time t The interpolated data at x k and x k+1 adjacent timestamps t k and t k+1 The original data at

[0106] For modalities with nonlinear time drift (such as IMU), the Kalman filter is used to estimate the time offset Δt, and the timestamp is dynamically adjusted through the state update equation, which is expressed as:

[0107]

[0108] in, K k is the Kalman gain, t ref is the reference time, t imu,k IMU timestamp.

[0109] In the above step S2, if Figure 3As shown, for image features, a pre-trained CNN model (such as ResNet or EfficientNet) is used to extract features from images captured by the image acquisition device to obtain high-level semantic feature vectors;

[0110] Perform global average pooling on the output of the intermediate layer or fully connected layer of the CNN model to compress the three-dimensional feature map C×H×W into a one-dimensional feature vector C×1. The calculation formula is:

[0111]

[0112] in, F c,i,j The cth channel of the feature map is at position (i,j) The activation value at f img is the compressed image feature vector.

[0113] In the above step S2, if Figure 4 As shown in the figure, for point cloud features, the PointNet or PointNet++ network is used to extract geometric features from the point cloud data collected by the lidar, specifically including:

[0114] Represent the original point cloud data as an N×3 matrix, where N is the number of point clouds and each row contains three-dimensional coordinates (x, y, z);

[0115] The multi-layer perceptron MLP is used to embed features of each point to generate a high-dimensional feature vector, and the feature space is aligned and transformed through the T-Net network. The calculation formula of the transformation matrix T is:

[0116]

[0117] in, P is the original point cloud matrix, P' is the transformed point cloud matrix, MLP θ is a multilayer perceptron with parameter θ;

[0118] Use maximum pooling to aggregate the features of all points to generate a global point cloud feature vector f lidar , expressed as:

[0119]

[0120] in, f n is the eigenvector of the nth point, R D is the feature dimension.

[0121] In the above step S2, if Figure 5As shown, for the IMU features, the accelerometer data a( t )=[ a x ( t ), a y ( t ), a z ( t )] T and gyroscope data oh ( t )=[ oh x ( t ), oh y ( t ), oh z ( t )] T , perform the following processing:

[0122] Use complementary filter or Kalman filter to denoise the raw data and estimate attitude angles, including pitch angle i , roll angle ϕ , yaw angle ψ , the update formula of its complementary filter is:

[0123]

[0124] in, i est is the estimated value, which represents the estimation of a parameter or state at the current moment. α is the learning rate or weighting factor that controls the balance between new information and old estimates, oh y ( t ) is the angular velocity, Δ t is the sampling time interval, a x ( t ) is the acceleration, ||a( t )|| is the acceleration modulus;

[0125] Time-integrate the gyroscope data to generate the rotation quaternion q(t) = [q0(t), q1(t), q2(t), q3(t)] T , its quaternion differential equation is:

[0126]

[0127] Where ⊗ is the quaternion multiplication operator.

[0128] The attitude angle, rotation quaternion and acceleration modulus||a( t )||Combined into IMU features f imu :

[0129]

[0130] Through the above steps, the high-level semantic features of the camera image, the geometric features of the lidar point cloud, and the motion state features of the IMU are extracted respectively, providing high-quality, structured feature representation for subsequent multimodal feature fusion.

[0131] In the above step S2, if Figure 6 As shown in the figure, when the graph neural network is used to fuse the extracted image features, point cloud features and IMU features, it includes:

[0132] Define the graph structure and treat the features of each modality as a node in the graph, where image features correspond to node A, point cloud features correspond to node B, and IMU features correspond to node C. The edges between nodes represent the correlation between modalities, and the edge weights are determined by the correlation between modalities.

[0133] Initialize node features. The initial features of each node are generated by the feature extractor of the corresponding modality. The features of node A are extracted by the CNN model, the features of node B are extracted by the PointNet or PointNet++ network, and the features of node C are generated by the complementary filter or Kalman filter.

[0134] The node features are updated through the message passing mechanism, which is expressed as the following formula:

[0135]

[0136] in, It is l Nodes in the layer i Features, is a node i The set of neighbor nodes of W and is a learnable weight matrix, s is the activation function;

[0137] After multiple rounds of message passing, the global pooling operation is used to aggregate the features of all nodes into a global feature vector, which is the output of multimodal fusion.

[0138] In the above step S3, if Figure 7 As shown, when the image features f img , point cloud features f lidar and IMU characteristicsf imu After fusion, the fused multimodal features are generated f fused :

[0139] The fused multimodal features f fused As visual input, the user's natural language instructions U As language input, construct the input pair of the visual language model (such as Qwen) f fused , U );

[0140] Use a visual encoder (such as ViT or CLIP) to process the visual input f fused Encode and generate visual feature vector f vis , expressed as the following formula:

[0141] f vis =Vis enc ( f fused ; i vis )

[0142] in, i vis are the visual encoder parameters;

[0143] Use target detection algorithms (such as YOLOv8 or Faster R-CNN) to locate the target frame and classify the target to generate target detection results; combine visual feature vectors f vis And the target detection results are used to generate a semantic map of the environment through spatial relationship modeling.

[0144] like Figure 8 As shown, the language encoder is used to encode the language input U to generate a language feature vector f lang , expressed as the following formula:

[0145] f lang =Lang enc ( U ; i lang )

[0146] in, i lang is the language encoder parameter;

[0147] User natural language commands U(such as "Take me to the kitchen") into a sequence of tokens U =[ u 1, u 2,…, uL ], and generate the language feature vector f through the language encoder lang .

[0148] Visual features via cross-attention mechanism f vis and language features f lang Interaction between them to generate cross-modal features f cross , expressed as the following formula:

[0149]

[0150] in, α i,j Represents the attention weight, which is used to measure the language feature vector f lang The i elements and visual feature vectors f vis,j The j The degree of correlation between elements, f vis,j for The first j elements, W Q 、W K 、 W V is the learnable weight matrix, d is the scaling factor, T Represents the transpose of the matrix, that is, ( W Q f lang ) T Representation matrix W Q f lang The transpose of .

[0151] The above formula measures the correlation between language features and visual features through dot product operations. The softmax function converts these correlations into probability distributions, namely attention weights, which can accurately capture the complex interactive relationship between visual and language modalities. The generated attention weights are used to perform weighted summation on visual features, thereby integrating language information into visual features and generating cross-modal features. This can effectively integrate the information of the two modalities, so that the generated cross-modal features can contain semantic information of both vision and language. And by adjusting W Q 、W K 、W V Using these matrices, the model can learn the optimal mapping between features from different modalities, thereby improving the quality of cross-modal features. The basic idea of ​​the cross-attention mechanism can be easily extended to handle more modalities. For example, if audio modality needs to be added, simply introduce the corresponding query, key, and value matrices, calculate attention weights in a similar manner, and perform feature fusion, making the model highly scalable.

[0152] Using multi-layer Transformer modules to transform cross-modal features f cross Perform progressive enhancement and output semantic maps M sem , target location T The visual analysis results are expressed as the following formula:

[0153] f enhanced =Transformer N dec ( f cross ; i cross )

[0154] M sem =Decoder map ( f enhanced ; i map )

[0155] T =Detector( f enhanced ; i det )

[0156] A=Policy( f enhanced ; ipolicy )

[0157] in, f enhanced For the enhanced cross-modal feature vector, Transformer N dec To represent a module consisting of N layers of Transformer decoders, Decoder map is the semantic map decoder, Detector is the target detector, Policy is the policy generation module, i cross 、 i map 、 i det 、 i policy are the parameters of the cross-modal Transformer decoder, semantic map decoder, object detector, and policy generation module respectively;

[0158] Based on semantic map M sem, semantically annotate different areas in the environment (such as "road", "obstacle", "building", etc.); according to the target location T , mark specific targets (such as "pedestrians in front", "vehicles on the left", etc.) in the semantic map; combined with action strategy A, update the semantic map in real time to reflect dynamic changes in the environment.

[0159] In this embodiment, action strategy A includes:

[0160] Waypoint sequence: based on cross-modal features f cross , using A* algorithm or Dijkstra algorithm in semantic map M Search the optimal path in sem and generate a path point sequence M. The generated path point sequence M is expressed as {( x 1, y 1),( x 2, y 2),…,( xn , in )},in( xi , yes ) is the i The coordinates of the path points;

[0161] Obstacle avoidance rules: When a dynamic obstacle is detected, the path is adjusted using a local path replanning algorithm to generate a new pathpoint sequence M′. For example, when an obstacle is detected ahead, the obstacle avoidance rules may include: slowing down or stopping to avoid collision; adjusting the robot's posture to circumvent the obstacle; and selecting an appropriate obstacle avoidance path based on the distribution of obstacles.

[0162] Through the above steps, the effective fusion of image, point cloud and IMU features is achieved, and the semantic segmentation and target detection capabilities of the visual language model are utilized to generate high-precision semantic maps, target locations and action strategies, significantly improving the semantic understanding and task planning capabilities in complex environments.

[0163] This embodiment also includes: using a lightweight model to accelerate the visual language model and deploying the optimized visual language model on edge devices. For example, using TensorRT to accelerate the Qwen model to reduce inference latency, and deploying the optimized model on NVIDIA Jetson devices to achieve low-latency real-time control.

[0164] In practical applications, this embodiment proposes a robot obstacle avoidance and navigation method based on multimodal fusion (fusion of camera images, lidar point clouds and IMU data) and visual language models (such as Qwen). Through the visual language model, a deep combination of visual features and language features is achieved to complete semantic understanding of the environment and task planning, and a series of scientific and effective steps are adopted in the reasoning process.

[0165] The visual language model uses a visual encoder to perform semantic segmentation on images, accurately labeling object categories such as shelves, storage locations, and conveyor lines. It also uses object detection algorithms (such as YOLO or Faster R-CNN) to precisely identify the location and size of specific objects. This capability enables robots to clearly identify various objects in complex environments, avoiding misjudgments of the environment. For example, in a logistics warehouse, robots can accurately identify the specific location and layout of shelves, as well as the status of goods on conveyor lines, providing detailed environmental information for subsequent task execution.

[0166] Combining the results of semantic segmentation and object detection, the visual language model can generate a semantic map of the environment, detailing information such as the workshop layout and obstacle distribution. This semantic map not only includes the geometric information of the environment but also assigns specific semantic meaning to each area and object. For example, in a factory workshop, the semantic map can clearly show layout information such as the shelf on the left, the conveyor line on the right, and the buffer rack and conveyor line in the center of the workshop. This allows the robot to understand the environmental structure at a macro level, providing strong support for task planning and path navigation.

[0167] The language encoder parses user commands, quickly extracting key information and clarifying the task's target location and specific requirements. For example, for the command "Go to conveyor #1 to load a material basket," the language encoder accurately interprets the target location as "Conveyor Line #1." This ensures the robot accurately understands the user's intent and avoids task failures due to misinterpretation of the command.

[0168] The visual language model's ability to parse language commands enables it to flexibly adapt to various forms of natural language instructions. Users can issue tasks in clear and concise language, without having to follow specific formats or rules. For example, a user can say "deliver the goods to the warehouse door" or "go to the office to pick up documents," and the robot will accurately understand and execute the corresponding task, greatly improving the user experience and reducing operational difficulty.

[0169] Using a cross-attention mechanism, the visual language model can accurately match verbal instructions with semantic maps. For example, based on the semantic map's information that the "conveyor line" is located in the center of the workshop, combined with the verbal instruction "Go to conveyor line 1," the model can generate a path plan starting from the current location, bypassing obstacles, and following the passage to the conveyor line interface. This enables the robot to combine verbal instructions with the actual environment to generate a more reasonable and efficient path.

[0170] While the robot is performing a task, the environment may change, such as new obstacles appearing or the target moving. The cross-modal reasoning capabilities of the visual language model enable the robot to perceive environmental changes in real time and dynamically adjust its path based on this new information. For example, if the robot encounters a temporary obstacle on its way to the target, the model can quickly replan its path, bypassing the obstacle and continuing on its way, ensuring successful completion of the task.

[0171] During the decision-making phase, the visual language model outputs specific action strategies, including waypoint sequences and obstacle avoidance rules. For example, the waypoint sequence [(x1, y1), (x2, y2), (x3, y3)] provides the robot with a clear path to follow, while the obstacle avoidance rule "adjust the local path when a dynamic obstacle is detected" ensures that the robot can react promptly to avoid collisions. These specific action strategies provide detailed guidance for the robot's operations, enabling it to execute tasks according to the planned plan.

[0172] The decision-making process fully considers various factors in the actual scenario, such as obstacle distribution and target location. Through reasonable path planning and obstacle avoidance rules, the robot can complete tasks safely and efficiently in complex environments. For example, in the above example of "avoiding obstacles ahead and heading to the door," the generated waypoint sequence [(0, 0), (1, 1), (3, 2), (5, 0)] can guide the robot smoothly around the box and reach the door, improving the success rate of task execution.

[0173] Using lightweight models and hardware acceleration technology can effectively reduce the inference latency of visual language models. While maintaining model accuracy, lightweight models reduce the number of model parameters and computational complexity, thereby improving inference speed. Hardware acceleration technology utilizes specialized hardware devices (such as GPUs and TPUs) to accelerate the model, further shortening inference time. This enables robots to respond to environmental changes in real time and adjust their action strategies promptly.

[0174] In complex and dynamic environments, robots must operate with a high degree of stability and real-time performance. Through real-time optimization processing, visual language models ensure that robots can make quick and accurate decisions in rapidly changing environments. For example, in a logistics warehouse, goods and personnel are frequently moved, and robots need to perceive these changes and adjust their actions in real time. Real-time optimization processing enables robots to maintain stable operation in this dynamic environment, improving system reliability and stability.

[0175] In summary, the multimodal fusion robot obstacle avoidance and navigation method based on a visual language model in this embodiment significantly improves the robot's obstacle avoidance and navigation capabilities in complex environments through precise semantic understanding, intelligent language command parsing, cross-modal reasoning fusion, scientific decision generation, and real-time optimization processing. It provides strong support for the widespread application and intelligent development of robots, and has broad market prospects and important practical value.

[0176] like Figure 9 As shown, this embodiment also provides a robot obstacle avoidance and navigation system based on multimodal fusion and visual language model, including the following modules:

[0177] Data acquisition module: used to acquire camera images, lidar point clouds and IMU data in real time.

[0178] Preprocessing module: performs time synchronization and format conversion on multimodal data.

[0179] Multimodal fusion module: extracts features of data from different modalities and fuses them through the intermediate layer.

[0180] Visual language model module: responsible for image analysis and semantic reasoning.

[0181] Decision generation module: Generates action strategies based on language instructions.

[0182] Real-time optimization module: uses lightweight models and hardware acceleration technology to improve real-time performance.

[0183] Action execution module: converts the strategy into control signals and drives the robot.

[0184] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0185] The above embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and replacements made by technicians in this field on the basis of the present invention fall within the scope of protection required by the present invention.

Claims

1. A robot obstacle avoidance and navigation method based on multimodal fusion and visual language model, characterized in that: The following steps are involved: Collect different modal data in real time through multiple sensors, and perform time synchronization and normalization processing; Extract features from different modal data and fuse them through an intermediate layer, where the intermediate layer is a graph neural network that fuses image features, point cloud features, and IMU features to generate a unified perception output; The fused multimodal data is input into the visual language model, and the semantic segmentation and target detection results of the model are used to generate a semantic map of the environment. The visual input is encoded using a visual encoder to generate a visual feature vector, and the language input is encoded using a language encoder to generate a language feature vector. The interaction between visual features and language features is achieved through the cross-attention mechanism to generate cross-modal features. f cross , expressed as the following formula: in, α i,j Represents the attention weight, which is used to measure the language feature vector f lang The i elements and visual feature vectors f vis,j The j The degree of correlation between elements, f vis,j is the first j elements, W Q 、W K 、W V is the learnable weight matrix, d is the scaling factor, T Represents the transpose of a matrix; Use multi-layer Transformer modules to process cross-modal features and generate visual analysis results; Combine natural language instructions and visual analysis results to generate action strategies; The generated action strategy is converted into a control signal that can be executed by the robot to achieve closed-loop control.

2. The method according to claim 1, wherein: When performing time synchronization processing, the image acquisition device, lidar and IMU device are coarsely aligned using PPS pulse signals or shared clock sources; For modal data that are not strictly synchronized, a timestamp-based interpolation or resampling method is used to align discrete time points using a linear interpolation formula, which is expressed as: in, x(t) For time t The interpolated data at x k and x k+1 adjacent timestamps t k and t k+1 The original data at For modes with nonlinear time drift, the Kalman filter is used to estimate the time offset Δt, and the timestamp is dynamically adjusted through the state update equation, which is expressed as: in, K k is the Kalman gain, t ref is the reference time, t imu,k IMU timestamp.

3. The method according to claim 1, wherein: For image features, a pre-trained CNN model is used to extract features from images captured by the image acquisition device to obtain high-level semantic feature vectors; Perform global average pooling on the output of the intermediate layer or fully connected layer of the CNN model to compress the three-dimensional feature map C×H×W into a one-dimensional feature vector C×1. The calculation formula is: in, F c,i,j The cth channel of the feature map is at position (i,j) The activation value at f img is the compressed image feature vector.

4. The method according to claim 1, wherein: For point cloud features, the PointNet or PointNet++ network is used to extract geometric features from the point cloud data collected by the lidar, specifically including: Represent the original point cloud data as an N×3 matrix, where N is the number of point clouds and each row contains three-dimensional coordinates (x, y, z); The multi-layer perceptron MLP is used to embed features of each point to generate a high-dimensional feature vector, and the feature space is aligned and transformed through the T-Net network. The calculation formula of the transformation matrix T is: in, P is the original point cloud matrix, P' is the transformed point cloud matrix, MLP θ is a multilayer perceptron with parameter θ; Use maximum pooling to aggregate the features of all points to generate a global point cloud feature vector f lidar , expressed as: in, f n is the eigenvector of the nth point, R D is the feature dimension.

5. The method according to claim 1, wherein: For IMU features, the accelerometer data a( t )=[ a x ( t ), a y ( t ), a z ( t )] T and gyroscope data ω ( t )=[ ω x ( t ), ω y ( t ), ω z ( t )] T , perform the following processing: Use complementary filter or Kalman filter to denoise the raw data and estimate attitude angles, including pitch angle θ , roll angle ϕ , yaw angle ψ , the update formula of its complementary filter is: in, θ est is the estimated value, which represents the estimation of a parameter or state at the current moment. α is the learning rate or weighting factor that controls the balance between new information and old estimates, ω y ( t ) is the angular velocity, Δ t is the sampling time interval, a x ( t ) is the acceleration, ||a( t )|| is the acceleration modulus; Time-integrate the gyroscope data to generate the rotation quaternion q(t) = [q0(t), q1(t), q2(t), q3(t)] T , its quaternion differential equation is: Among them, ⊗ is the quaternion multiplication operator; The attitude angle, rotation quaternion and acceleration modulus||a( t )||Combined into IMU features f imu .

6. The method according to claim 1, characterized in that When using graph neural networks to fuse the extracted image features, point cloud features, and IMU features, the following steps are involved: Define the graph structure and treat the features of each modality as a node in the graph, where image features correspond to node A, point cloud features correspond to node B, and IMU features correspond to node C. The edges between nodes represent the correlation between modalities, and the edge weights are determined by the correlation between modalities. Initialize node features. The initial features of each node are generated by the feature extractor of the corresponding modality. The features of node A are extracted by the CNN model, the features of node B are extracted by the PointNet or PointNet++ network, and the features of node C are generated by the complementary filter or Kalman filter. The node features are updated through the message passing mechanism, which is expressed as the following formula: in, It is l Nodes in the layer i Features, is a node i The set of neighbor nodes of W and is a learnable weight matrix, σ is the activation function; After multiple rounds of message passing, the global pooling operation is used to aggregate the features of all nodes into a global feature vector, which is the output of multimodal fusion.

7. The method according to any one of claims 1 to 6, characterized in that: In the image features f img , point cloud features f lidar and IMU characteristics f imu Perform fusion to generate fused multimodal features f fused : The fused multimodal features f fused As visual input, the user's natural language instructions U As language input, construct the input pair of the visual language model ( f fused , U ); Use visual encoder to process visual input f fused Encode and generate visual feature vector f vis , expressed as the following formula: f vis =All enc ( f fused ; θ vis ) in, θ vis are the visual encoder parameters; The target detection algorithm is used to locate the target frame and classify the target to generate the target detection result; combined with the visual feature vector f vis And the target detection results are used to generate a semantic map of the environment through spatial relationship modeling.

8. The method according to claim 7, wherein: Use the language encoder to encode the language input U and generate the language feature vector f lang , expressed as the following formula: f lang =Just enc ( U ; θ lang ) in, θ lang The language encoder parameters.

9. The method according to claim 8, characterized in that: Using multi-layer Transformer modules to transform cross-modal features f cross Perform progressive enhancement and output semantic maps M sem , target location T The visual analysis results are expressed as the following formula: f enhanced =Transformer N dec ( f cross ; θ cross ) M sem =Decoder map ( f enhanced ; θ map ) T =Detector( f enhanced ; θ det ) A=Policy( f enhanced ; θ policy ) in, f enhanced For the enhanced cross-modal feature vector, Transformer N dec To represent a module consisting of N layers of Transformer decoders, Decoder map is the semantic map decoder, Detector is the target detector, Policy is the policy generation module, θ cross 、 θ map 、 θ det 、 θ policy are the parameters of the cross-modal Transformer decoder, semantic map decoder, object detector, and policy generation module respectively; Based on semantic map M sem, semantically annotate different areas in the environment; according to the target location T , mark specific targets in the semantic map; combined with action strategy A, update the semantic map in real time to reflect dynamic changes in the environment.

10. The method according to claim 9, characterized in that: The action strategy A includes: Inferring target locations and generating waypoint sequences from semantic maps in navigation tasks: Based on cross-modal features f cross , using A* algorithm or Dijkstra algorithm in semantic map M Search the optimal path in sem and generate a path point sequence M. The generated path point sequence M is expressed as {( x 1, y 1),( x 2, y 2),…,( xn , yn )},in( xi , yi ) is the i The coordinates of the path points; The obstacle avoidance rule formed by using the dynamic window method to plan the path in the obstacle avoidance task is: when a dynamic obstacle is detected, the path is adjusted through the local path replanning algorithm to generate a new path point sequence M′.

Images