Robot navigation obstacle avoidance method and system based on multiple sensors

Abstract

The invention relates to the technical field of robots, in particular to a robot navigation obstacle avoidance method and system based on multiple sensors. The method comprises the following steps: collecting multi-modal data, wherein the multi-modal data comprises point cloud data, a visible light image and an infrared light image; inputting the multi-modal data into a three-dimensional perception model for target detection and semantic segmentation to obtain a first prediction result, the first prediction result comprising a semantic segmentation map; inputting the visible light image into a two-dimensional perception model for segmentation to obtain a second prediction result; first mapping is carried out on the first prediction result and the second prediction result, and when first mapping is carried out, if two kinds of semantic information exist in corresponding targets in the first prediction result and the second prediction result, the semantic information with the higher confidence coefficient score serves as the semantic information of the corresponding targets. And obtaining a semantic map for navigation and obstacle avoidance. According to the method, the semantic map for improving the robot obstacle avoidance decision accuracy can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of robotics, and more particularly to a multi-sensor based robot navigation and obstacle avoidance method and system. Background Art

[0002] With the development of science and technology, robots are widely used in various fields such as life services and industrial production. However, during the movement of robots, they will inevitably encounter obstacles on the road. Therefore, the issue of how robots can effectively avoid obstacles while driving has attracted widespread attention. In the existing technology, a single sensor is usually used for perception and obstacle avoidance. For example, a pure visual sensor or a pure lidar is used to perceive the surrounding environment, and then the robot avoids obstacles based on the perceived information. However, this obstacle avoidance method based on a single sensor is generally affected by factors such as feature point matching, lighting or complex environment, resulting in the inability to accurately identify untrained obstacles, resulting in poor obstacle detection effect, and lack of corresponding semantic information, thus failing to provide a reliable basis for the robot's obstacle avoidance decision-making.

[0003] In this regard, how to accurately obtain the semantic map of the robot's surrounding environment to provide a reliable basis for obstacle avoidance decisions is a technical problem that needs to be solved urgently. Summary of the Invention

[0004] In order to solve the above-mentioned technical problem of being unable to accurately identify obstacles during obstacle avoidance and having no semantic information of the obstacles, the present invention provides solutions in the following aspects.

[0005] In a first aspect, the present invention provides a multi-sensor based robot navigation and obstacle avoidance method, comprising: collecting multimodal data, wherein the multimodal data includes point cloud data, visible light images, and infrared light images; inputting the multimodal data into a three-dimensional perception model for target detection and semantic segmentation to obtain a first prediction result, wherein the first prediction result includes a semantic segmentation map; inputting the visible light image into a two-dimensional perception model for segmentation to obtain a second prediction result; performing a first mapping on the first prediction result and the second prediction result. When performing the first mapping, if there are two types of semantic information for corresponding targets in the first prediction result and the second prediction result, the semantic information with a higher confidence score is used as the semantic information of the corresponding target to obtain a semantic map for navigation and obstacle avoidance.

[0006] Furthermore, the first prediction result also includes a bounding box, and the multimodal data is input into a three-dimensional perception model for target detection and semantic segmentation, including: extracting features of the multimodal data and fusing them; inputting the fused features into a backbone feature network for extraction to obtain extraction results; and inputting the extraction results into a detection head for obtaining the bounding box and a semantic segmentation head for obtaining the semantic segmentation map.

[0007] Furthermore, the features of the multimodal data are extracted and then fused, including: obtaining a first BEV feature based on the point cloud data, obtaining a second BEV feature based on the visible light image, and obtaining a third BEV feature based on the infrared light image; based on the channel latitude, the first BEV feature, the second BEV feature, and the third BEV feature are spliced ​​together, and then fused through multiple preset structures.

[0008] Furthermore, fusion is performed through multiple preset structures, including: fusion is performed through multiple preset structures composed of Conv2D, BatchNorm, and PRelu.

[0009] Furthermore, before obtaining the first BEV feature, the method further includes: filtering out point cloud data within a preset range from the point cloud data as point cloud data used for obtaining the first BEV feature.

[0010] Furthermore, obtaining a first BEV feature based on the point cloud data includes: obtaining the first BEV feature based on a PointPillar algorithm; obtaining a second BEV feature based on the visible light image includes: obtaining an image feature based on a ResNet34 model, and mapping the image feature to obtain the second BEV feature.

[0011] Furthermore, after obtaining the semantic map, the method further includes: inputting the semantic map into a trained decision model to make a decision, and obtaining an obstacle avoidance decision result, wherein the obstacle avoidance decision result includes an obstacle avoidance path.

[0012] Furthermore, inputting the semantic map into the decision model includes: unifying all information in the semantic map into structured data, and then inputting the structured data into the decision model.

[0013] Furthermore, the training method of the decision model includes: generating simulated data and collecting real data in a real physical environment; using the simulated data to train a preset neural network to obtain a base model; using the real data to test the base model, and selecting first target data from the real data based on the test results; training the base model based on the first target data and the simulated data, and stopping training in response to the accuracy of the current model tested in the real data being greater than a preset threshold; and then using the second target data for fine-tuning to obtain the decision model.

[0014] In a second aspect, the present invention provides a multi-sensor based robot navigation and obstacle avoidance system, comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a multi-sensor based robot navigation and obstacle avoidance method according to any one of the first aspects is implemented.

[0015] The beneficial effects of the present invention are as follows: By utilizing multimodal data to predict objects and semantic information in the robot's surrounding environment, the present invention improves the accuracy of object detection and semantic segmentation compared to traditional single-model, single-modality perception. By mapping the more accurate and comprehensive semantic information obtained from a two-dimensional perception model into a semantic map obtained from a three-dimensional perception model, a more comprehensive and precise semantic map is obtained, providing a reliable and accurate basis for the robot's obstacle avoidance decisions. By converting the resulting semantic map into structured data and inputting it into the decision-making model, the efficiency and accuracy of decision-making can be improved, and the system has greater adaptability to unusual and unexpected situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart schematically illustrating a multi-sensor based robot navigation and obstacle avoidance method according to an embodiment of the present invention;

[0017] Figure 2 is a flowchart schematically illustrating target detection and semantic segmentation according to an embodiment of the present invention;

[0018] Figure 3 is a structural diagram schematically illustrating a three-dimensional perception model according to an embodiment of the present invention;

[0019] Figure 4 is a schematic diagram schematically illustrating mapping image features according to an embodiment of the present invention;

[0020] Figure 5 FIG. 4 is a block diagram schematically illustrating a multi-sensor based robot navigation and obstacle avoidance system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0022] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Figure 1 FIG. 4 is a flow chart schematically illustrating a multi-sensor based robot navigation and obstacle avoidance method according to an embodiment of the present invention.

[0024] In order to solve the technical problems that the existing technology using a single sensor for perception has poor detection effect on irregular obstacles (untrained target obstacles, such as ice cream cones, stone pillars, etc.) and lacks semantic information, in the first aspect, the present invention provides a robot navigation and obstacle avoidance method based on multiple sensors, which is applicable to mobile devices such as service robots, autonomous vehicles, AGVs (Automated Guided Vehicles, etc.). Figure 1 As shown, the method of the present invention includes:

[0025] S1. Collect multimodal data.

[0026] In this embodiment, multimodal data includes point cloud data, visible light images, and infrared images. Specifically, point cloud data can be collected using a lidar, visible light images can be collected using a visible light camera, and infrared images can be collected using an infrared camera. The visible light camera and infrared camera can be fixed with a specific structure at the factory and aligned so that their fields of view approximately overlap. The lidar and visible light camera can be calibrated for internal and external parameters using calibration software.

[0027] In actual applications, there may be inconsistencies in the acquisition frequencies of devices. For example, the sampling rate of the lidar is 10FPS (Frames Per Second), and the sampling rates of the visible light camera and infrared camera are 30FPS. In addition, the lidar is not synchronized with the visible light camera and infrared camera in terms of timing. To this end, the acquisition of point cloud data and images (including visible light images and infrared images) can be synchronized in timing with a certain delay, and downsampled to align the sampling rate of the image with the sampling rate of the point cloud data. Finally, data is acquired at a sampling rate of 10FPS.

[0028] S2. Input the multimodal data into a three-dimensional perception model for semantic segmentation and target detection to obtain a first prediction result.

[0029] In this embodiment, the three-dimensional perception model is mainly composed of an Open Occupancy model; the first prediction result includes the position information (corresponding to the bounding box) and semantic information (corresponding to the semantic segmentation map) of all targets.

[0030] In this embodiment, the targets referred to include four types, namely road targets (drivable road surfaces, road lane lines, etc.), surrounding environment specific dynamic targets (pedestrians, motor vehicles, non-motor vehicles, etc.), surrounding environment specific sign targets (traffic lights, warning signs, etc.), and surrounding environment non-specific obstacle targets (roadblocks, road ice cream cones, stone piers, etc.).

[0031] Compared to single-sensor perception solutions, this invention integrates not only point cloud data and visible light images, but also infrared images. By integrating data from multiple sensors, the overall network accuracy and generalization capabilities are improved, thereby enhancing the robot's ability to accurately identify obstacles in extreme and complex environments, providing a reliable basis for subsequent obstacle avoidance decisions.

[0032] Specifically, if Figure 2 As shown in Figure 2, the process of target detection and semantic segmentation based on the 3D perception model is as follows:

[0033] S201. Perform feature fusion based on multimodal data.

[0034] Specifically, combined Figure 3 The point cloud data is voxelized (dividing the continuous point cloud into small cubes) and features are extracted to obtain voxelized feature data. The voxelized feature data is input into a 3D convolutional network to obtain convolutional feature data (higher-level, more abstract features). This is then downsampled to obtain the first BEV (Bird's-Eye View) feature.

[0035] During voxelization and feature extraction, this paper refers to the PointPillar (Point-based Pillars) algorithm, but only uses four elements: X, Y, Z coordinates and reflectivity. By using only these four elements, the dimensionality of the input data is reduced, thereby reducing the amount of calculation and improving computational efficiency.

[0036] In one embodiment, before voxelizing the point cloud data, the method of the present invention further includes: selecting point cloud data within a preset range from the collected point cloud data as the point cloud data required for subsequent processing. Specifically, point cloud data within the Z-axis range of -5 to 5 meters, the X-axis range of -10 to 10 meters, and the Y-axis range of -2 to 2 meters from the collected point cloud data are used as input for the 3D perception model. By selecting point cloud data within the preset range as input, not only can detection accuracy be ensured, but also the amount of computation can be reduced, thereby improving the efficiency and accuracy of feature extraction.

[0037] At the same time, the visible light image (2D image) is input into a feature extraction network (in this embodiment, a ResNet34 model pre-trained on ImageNet) for feature extraction to obtain image features, which are then mapped to obtain the second BEV features. Similarly, for the infrared image (2D image), the infrared image is also input into a ResNet34 (residual convolutional neural network) model pre-trained on ImageNet (a large-scale visual database) to obtain image features, which are then mapped to obtain the third BEV features.

[0038] In one embodiment, Figure 4 As shown in the figure, the process of mapping image features (i.e., the second mapping) is as follows: first, the correspondence between point cloud data and images (visible light images, infrared images) is obtained through pre-calibrated internal parameters (such as camera focal length, etc.) and external parameters (camera position and posture in the world coordinate system); since point cloud data to 3D BEV features (corresponding BEV features) can be pre-acquired through voxelization and downsampling, for each position on the 3D BEV feature map, multiple mappings can be found on the 2D image features (corresponding image features), and finally, by weighting the corresponding positions, the 2D image features can be mapped to the 3D BEV feature map (the height in the 3D BEV feature map is compressed to 1).

[0039] Furthermore, the first BEV feature, the second BEV feature, and the third BEV feature are fused. Specifically, based on the second dimension (C channel latitude), the first BEV feature, the second BEV feature, and the third BEV feature are spliced, and then fused through three Conv2D+BatchNorm+PRelu structures (i.e., the preset structure) to obtain the fused features. Among them, Conv2D is a two-dimensional convolution, BatchNorm is batch normalization, and PRelu is a parameterized Relu activation function. Through the Conv2D+BatchNorm+PRelu structure, the generalization ability of the three-dimensional perception model can be improved, thereby improving the accuracy of subsequent target detection and semantic segmentation.

[0040] S202: Input the fused features into the backbone feature network for extraction to obtain the extraction results.

[0041] In this embodiment, the backbone feature network used is the MobileNet-V2 network (a lightweight convolutional neural network), which can improve the accuracy of feature extraction compared to the original VGG-like (Visual Geometry Group-like) network.

[0042] S203: Input the extraction results into the target detection head and the semantic segmentation head respectively.

[0043] Specifically, in the target detection task, the extraction results are input into the single-stage 3D detection head to obtain the 3D detection bounding box; at the same time, in the semantic segmentation task, the extraction results are upsampled (fused feature upsampling, the original voxel features are fused into the BEV features for upsampling, and the BEV features referred to are the second BEV features and third BEVs generated by mapping the 2D image to the 3D space), and then input into the semantic segmentation head to obtain a 3D semantic segmentation map.

[0044] In this embodiment, the training method of the 3D perception model is as follows: in the first stage, the three branches are trained separately (i.e. Figure 3 The first stage is to use the network trained in the first stage to train the overall fusion network; during training, the parameters of the network trained in the first stage are fixed, and then only the untrained networks in the overall network (fusion network, backbone feature network, detection head and semantic segmentation head network) are trained to complete the training of the 3D perception model.

[0045] During the training process, the data used include open source data and real data (data collected in actual scenarios and have been labeled).

[0046] Since the amount of information in the 3D semantic segmentation map is relatively large, these semantic information can be converted into the form of target frames (including dynamic target frames and static target frames) to facilitate subsequent mapping.

[0047] S3. Input the visible light image into a two-dimensional perception model for semantic segmentation to obtain a second prediction result.

[0048] In this embodiment, the two-dimensional perception model is a YOLOE model, and the second prediction result includes semantic information of all targets (2D result graph).

[0049] By inputting visible light images into the two-dimensional perception model, more refined and accurate semantic information of all targets in the overall environment can be obtained, namely, lane lines on the road, categories of dynamic targets (distinguishing between pedestrians, motor vehicles, and non-motor vehicles), status of traffic lights (red, yellow, green), categories of warning signs (speed limit, no driving, left turn, right turn, etc.), and other relevant semantic information of targets.

[0050] By using the YOLOE model (this model can handle more categories of targets, is not limited to pre-set fixed categories, and also has certain detection and segmentation capabilities when facing new or unseen categories), the accuracy, generalization ability and versatility of detection and segmentation can be improved.

[0051] S4. Perform a first mapping on the first prediction result mapping and the second prediction result to obtain a semantic map for navigation and obstacle avoidance.

[0052] In one embodiment, the 2D result map output by the two-dimensional perception model and the 3D semantic segmentation map output by the three-dimensional perception model are mapped to each other (i.e., the first mapping). Specifically, the point cloud information in the semantic map is projected (mapped) onto the 2D result map output by the two-dimensional perception model through the previously pre-calibrated internal and external parameters, and then the semantic information of which 2D areas and which targets can be reprojected onto the 3D semantic segmentation map can be determined through the 2D result map, and then these areas, targets, and semantic information that can be reprojected are projected onto the 3D semantic segmentation map. During the projection process, for those areas without detailed semantic information on the 3D semantic segmentation map, the semantic information of the 2D result map is used to supplement it; if semantic information already exists on the 3D semantic segmentation map (the original 3D semantic segmentation map), that is, a target has semantic information in both the 2D result map and the 3D semantic segmentation map, then the semantic information with a higher confidence score is selected as the final semantic information of the target. The semantic map obtained after mapping is the semantic map that is ultimately used for obstacle avoidance decision-making. By using semantic information with higher confidence scores as the semantic information of the target in the semantic map, the reliability of the semantic map is further improved. It is understood that the output results of the 3D perception model and the 2D perception model also include the confidence scores corresponding to the segmented regions or semantic information.

[0053] In one embodiment, the confidence score used when comparing the confidence scores of two types of semantic information may be the average of the confidence scores of all the same categories in the segmented regions.

[0054] In an optional embodiment, the 2D result image output by the two-dimensional perception model can also be directly mapped to the 3D semantic segmentation map output by the three-dimensional perception model. For targets with semantic information that already exists in the 3D semantic segmentation map, semantic information with a higher confidence score is selected as the semantic information of the target in the final semantic map.

[0055] In the prior art, the semantic segmentation map obtained by the three-dimensional perception model is usually directly applied to the robot's obstacle avoidance navigation. However, the recognition ability of the semantic map obtained by the three-dimensional perception model is limited. It can usually only recognize a certain type of target (for example, only 10 types of targets), and there may be a situation where the target lacks semantic information. In response to this, the present invention obtains more refined and accurate semantic information of the surrounding environment by adding a two-dimensional perception model (for example, it can recognize 100 types of targets and corresponding semantic information), and maps this semantic information to the semantic map of the three-dimensional perception model. This can solve the problem that the semantic map obtained by the three-dimensional perception model can only recognize a limited number of target types and the semantic information accuracy is insufficient. It gives more accurate semantic information to all targets and enriches the target and semantic information of the perceived scene, thereby improving the accuracy and reliability of subsequent obstacle avoidance decisions based on the semantic map.

[0056] In summary, compared with the perception solution of a single sensor, the present invention uses multi-model and multi-modal perception technology, which can make the overall detection accuracy higher and the generalization ability stronger (it can predict irregular obstacles, such as ice cream cones, etc.), thereby providing a more accurate and reliable basis for the subsequent robot's obstacle avoidance decision-making.

[0057] In one embodiment, existing obstacle avoidance strategies are relatively fixed and difficult to meet the needs of dynamically changing scenarios. Even if dynamic obstacle avoidance is used, the single input variable makes it difficult to adapt to rapid changes in the environment, making it difficult to achieve ideal obstacle avoidance results. To address this issue, the method of the present invention further includes: inputting the semantic map obtained in S4 into a trained decision model to obtain an obstacle avoidance decision result.

[0058] In this embodiment, the obstacle avoidance decision result includes driving speed, driving direction, obstacle avoidance path, etc.

[0059] Specifically, the semantic map (target box and multi-attribute label data) obtained previously is input into the reinforced neural network model (i.e., decision model) obtained by the semi-supervised PPO training method to obtain the obstacle avoidance decision result.

[0060] In this embodiment, the decision model training process is as follows: First, in a simulation environment, various normal (common everyday scenarios) and abnormal (rare or dangerous scenarios, such as road construction) surrounding environments are simulated, as well as the agent (intelligent agent, i.e., robot) behaviors and corresponding results. Then, these behaviors are aligned to generate a large number of supervised data sequences (i.e., simulated data). Simultaneously, real data from the real physical world (i.e., data collected by sensors in real scenarios) is collected and annotated.

[0061] Since the information in the semantic map is very large, it is not conducive to the subsequent use of the enhanced neural network model. Therefore, all the semantic information is unified into structured data (in the form of topological structure information + multi-attribute labels), and the data is sampled at 10Hz for T seconds (in this embodiment, T is set to 2) to form a segment of data (obtained by simple splicing), and then this segment of data is input into the enhanced neural network model for training and prediction.

[0062] Specifically, the structure of a piece of data is as follows:

[0063]

[0064]

[0065] By converting the collected data into the above-mentioned data and then using it for training, the stability and accuracy of the decision model can be improved and the computational burden of the system can be reduced, thereby improving the accuracy of the decision model in obstacle avoidance decisions.

[0066] Furthermore, the preset neural network is trained using simulation data (all information about the surrounding environment, the agent's behavior, and the corresponding results that can be obtained through simulation) to obtain a base model.

[0067] The base model is evaluated and tested with the labeled real data, and the first target data is filtered out from the real data according to the test results. In this embodiment, the first target data is a sample with a large difference between the test result and the real result (i.e., Badcase data). The first target data and the simulated data are then used to train the base model together to obtain the M_f1 model; the real data is continuously collected to iteratively train the M_f1 model until the accuracy of the M_fi (i-th iteration) model tested in all real data is greater than the preset threshold (in this embodiment, set to 90%); finally, the second target data (the real data that caused the robot to fail in obstacle avoidance) is combined to fine-tune the M_fi model to obtain the final decision model.

[0068] In an optional embodiment, corresponding simulation data may be selectively generated according to the pattern of the first target data, and then the base model may be trained using the simulation data and the simulation data used when training the neural network.

[0069] During training, the PPO algorithm (Proximal Policy Optimization) is used for supervised training. The present invention sets up a variety of positive reward functions and negative reward (penalty) functions. Positive reward functions include: obstacle distance function, path maintenance function, target maintenance function, drivable road overlap function, etc.; negative reward functions control: dynamic target collision penalty function, static obstacle collision penalty function, special sign penalty function, lane line overlap penalty function, etc.

[0070] Use of the decision-making model: The perception information results of 20 frames are simply assembled in sequence to obtain a piece of data, which is then input into the strategy model to obtain the obstacle decision result. The obstacle decision result includes information such as driving direction, driving speed, and re-planned obstacle avoidance path. This information is then transmitted to each controller to control and guide the robot to perform corresponding obstacle avoidance and path re-planning.

[0071] It can be understood that each time a decision is made, a piece of data is input into the decision model to obtain a decision result. That is, the robot makes a decision every two seconds, and the data collected in these two seconds is input each time a decision is made.

[0072] Compared with single variable input, the accuracy, flexibility and stability of obstacle avoidance decisions are improved by making predictions based on a rich variety of perception results (the semantic map obtained previously); by using big data supervision and simulation to train the decision model, the decision model can not only handle classic obstacle avoidance scenarios, but also solve some abnormal emergencies through generalization. Therefore, it has stronger adaptability and accuracy to abnormal emergencies and strong environmental adaptability.

[0073] Figure 5 Schematically shows a structural block diagram of a multi-sensor based robot navigation and obstacle avoidance system according to this embodiment.

[0074] In a second aspect, the present invention also provides a robot navigation and obstacle avoidance system based on multiple sensors. Figure 5 As shown, the system includes a processor and a memory, wherein the memory stores computer program instructions. When the computer program instructions are executed by the processor, a multi-sensor based robot navigation and obstacle avoidance method according to the first aspect of the present invention is implemented.

[0075] The system also includes other components well known to those skilled in the art, such as a communication interface, and their configuration and functions are known in the art, so they will not be described in detail here.

[0076] In the present invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, the computer-readable storage medium can be any suitable magnetic storage medium or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc., or any other medium that can be used to store the required information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible or connectable to a device. Any application or module described in the present invention can be implemented using computer-readable / executable instructions that can be stored or otherwise retained by such a computer-readable medium.

[0077] While several embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous modifications, variations, and alternatives will occur to those skilled in the art without departing from the concept and spirit of the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed in practicing the present invention.

Claims

1. A robot navigation and obstacle avoidance method based on multiple sensors, characterized in that: include: Collecting multimodal data, the multimodal data including point cloud data, visible light images, and infrared light images; Inputting the multimodal data into a three-dimensional perception model to perform target detection and semantic segmentation to obtain a first prediction result, wherein the first prediction result includes a semantic segmentation map; Inputting the visible light image into a two-dimensional perception model for segmentation to obtain a second prediction result; A first mapping is performed on the first prediction result and the second prediction result. When performing the first mapping, if there are two types of semantic information for the corresponding targets in the first prediction result and the second prediction result, the semantic information with a higher confidence score is used as the semantic information of the corresponding target to obtain a semantic map for navigation and obstacle avoidance.

2. The multi-sensor based robot navigation and obstacle avoidance method according to claim 1, characterized in that: The first prediction result also includes a bounding box, and the multimodal data is input into a 3D perception model for target detection and semantic segmentation, including: Extracting features of the multimodal data and fusing them; The fused features are input into the backbone feature network for extraction to obtain the extraction results; The extraction results are respectively input into a detection head for obtaining the bounding box and a semantic segmentation head for obtaining the semantic segmentation map.

3. The multi-sensor based robot navigation and obstacle avoidance method according to claim 2, characterized in that: Extracting features of the multimodal data and fusing them, including: obtaining a first BEV feature based on the point cloud data, obtaining a second BEV feature based on the visible light image, and obtaining a third BEV feature based on the infrared light image; Based on the channel latitude, the first BEV feature, the second BEV feature, and the third BEV feature are spliced ​​and then fused through multiple preset structures.

4. The multi-sensor based robot navigation and obstacle avoidance method according to claim 3, characterized in that: Fusion is performed through multiple preset structures, including: fusion through multiple preset structures composed of Conv2D, BatchNorm, and PRelu.

5. The multi-sensor based robot navigation and obstacle avoidance method according to claim 3, characterized in that: Before obtaining the first BEV feature, the method further includes: screening out point cloud data within a preset range from the point cloud data as point cloud data used for obtaining the first BEV feature.

6. The multi-sensor based robot navigation and obstacle avoidance method according to claim 3, characterized in that: Obtaining a first BEV feature based on the point cloud data includes: obtaining the first BEV feature based on a PointPillar algorithm; obtaining a second BEV feature based on the visible light image includes: obtaining an image feature based on a ResNet34 model, and performing a second mapping on the image feature to obtain the second BEV feature.

7. The multi-sensor based robot navigation and obstacle avoidance method according to claim 1, characterized in that: After obtaining the semantic map, the method further includes: inputting the semantic map into a trained decision model to make a decision, thereby obtaining an obstacle avoidance decision result, wherein the obstacle avoidance decision result includes an obstacle avoidance path.

8. The multi-sensor based robot navigation and obstacle avoidance method according to claim 7, characterized in that: Inputting the semantic map into a decision model includes: All information in the semantic map is unified into structured data and then input into the decision model.

9. The multi-sensor based robot navigation and obstacle avoidance method according to claim 7, characterized in that: The training method of the decision model includes: Generate simulated data and collect real data in real physical environments; Using the simulation data to train a preset neural network to obtain a base model; Testing the base model using the real data, and selecting first target data from the real data according to the test result; The base model is trained based on the first target data and the simulation data, and the training is stopped in response to the accuracy of the current model tested in the real data being greater than a preset threshold; then, the second target data is used for fine-tuning to obtain the decision model.

10. A multi-sensor based robot navigation and obstacle avoidance system, characterized in that: The invention comprises a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a multi-sensor based robot navigation and obstacle avoidance method according to any one of claims 1 to 9 is implemented.

Images