Multi-sensor fusion obstacle avoidance method and system

Abstract

The invention relates to the technical field of obstacle avoidance, and discloses a multi-sensor fusion obstacle avoidance method and system, and the method comprises the steps: determining an image of a target to be subjected to obstacle avoidance, and determining obstacle point cloud data according to the image of the target to be subjected to obstacle avoidance; and determining a reflective mask image based on the sunlight segmentation model, extracting a pixel value of the reflective mask image, and judging whether a reflective area exists or not according to the pixel value and the obstacle point cloud data. If the reflection area does not exist, determining a first obstacle map according to the obstacle point cloud data, if the reflection area exists, deleting the obstacle point cloud data according to the reflection area, determining radar point cloud data according to the ultrasonic radar, and fusing the light avoiding point cloud data and the radar point cloud data to obtain a first obstacle map; an obstacle avoidance path is determined based on the first obstacle map or the second obstacle map. According to the invention, the reliability of obstacle avoidance is improved by identifying the reflective area and fusing the light-avoiding point cloud data and the radar point cloud data.

Description

Technical Field

[0001] The present invention relates to the technical field of obstacle avoidance, and in particular, to an obstacle avoidance method and system based on multi-sensor fusion. Background Art

[0002] With the booming development of the robot industry, mobile robots have been widely popularized in many fields, and higher requirements have been put forward for the safety, obstacle detection, obstacle avoidance, and path planning of robots. Especially in complex scenarios, there are certain challenges in aspects such as obstacle detection and smooth movement.

[0003] When a robot performs obstacle detection, in order to accurately avoid obstacles, it needs to make a judgment based on the three-dimensional information of the three-dimensional environment around the robot. Currently, robots usually use a 3D camera (such as a binocular 3D camera) to detect obstacles through point cloud data. However, when there is sunlight reflection on the ground, noise points will be generated in the reflective area of the point cloud data, resulting in false detection or missed detection of obstacles. A single sensor cannot meet the detection requirements under complex lighting conditions.

[0004] Therefore, it is necessary to design an obstacle avoidance method and system based on multi-sensor fusion to solve the problems existing in the current technology. Summary of the Invention

[0005] In view of this, the present invention proposes an obstacle avoidance method and system based on multi-sensor fusion, aiming to solve the problem that when there is sunlight reflection on the ground, noise points will be generated in the reflective area of the point cloud data, resulting in false detection or missed detection of obstacles, and a single sensor cannot meet the detection requirements under complex lighting conditions.

[0006] On the one hand, the present invention proposes an obstacle avoidance method based on multi-sensor fusion, including: Obtaining a plurality of images to be avoided, processing the plurality of images to be avoided to determine a target image to be avoided, and determining obstacle point cloud data according to the target image to be avoided; Based on the target image to be avoided, determining a reflective mask image based on a sunlight segmentation model, extracting pixel values of the reflective mask image, and judging whether there is a reflective area according to the pixel values and the obstacle point cloud data; When there is no such reflective area, determining a first obstacle map according to the obstacle point cloud data; when there is such a reflective area, deleting the obstacle point cloud data according to the reflective area to obtain light-avoiding point cloud data, determining radar point cloud data according to an ultrasonic radar, fusing the light-avoiding point cloud data and the radar point cloud data to determine target obstacle point cloud data, and determining a second obstacle map according to the target obstacle point cloud data; Determining an obstacle avoidance path based on the first obstacle map or the second obstacle map.

[0007] Further, when processing a plurality of the images to be obstacle-avoided to determine the target image to be obstacle-avoided, it includes: Preprocessing the plurality of the images to be obstacle-avoided, where the preprocessing includes denoising and geometric correction; Extract feature points from the plurality of preprocessed images to be obstacle-avoided according to the Speeded Up Robust Features (SURF) algorithm, match the extracted feature points using the Random Sample Consensus (RANSAC) algorithm, and determine the image correlation data between the plurality of preprocessed images to be obstacle-avoided; Based on the image correlation data, register the plurality of preprocessed images to be obstacle-avoided, and merge the registered plurality of images to be obstacle-avoided according to multi-band fusion to determine the target image to be obstacle-avoided.

[0008] Further, when determining the reflective mask image based on the sunlight segmentation model according to the target image to be obstacle-avoided, it includes: Obtain an image dataset, divide the image dataset into a training set and a test set, use cross-validation and grid search to find the model parameters of a neural network model, and establish a neural network model; Fit the neural network model according to the training set, and adjust the hyperparameters of the neural network model according to the root mean square error of the test set; Substitute the test set into the adjusted neural network model and calculate the accuracy rate. When the accuracy rate reaches the preset accuracy rate threshold, determine the reflective mask image according to the target image to be obstacle-avoided.

[0009] Further, when extracting the pixel values of the reflective mask image and determining whether there is a reflective area according to the pixel values and the obstacle point cloud data, it includes: Extract all the point cloud data points corresponding in the obstacle point cloud data, extract all the mask pixel points corresponding to the reflective mask image, and determine the pixel values of all the mask pixel points; When the pixel value of the mask pixel point corresponding to the point cloud data point is 1, generate a reflective mark for the point cloud data point; When the pixel value of the mask pixel point corresponding to the point cloud data point is 0, generate a non-reflective mark for the point cloud data point; If the reflective mark is recognized, it is determined that there is a reflective area; If the reflective mark is not recognized, it is determined that there is no reflective area.

[0010] Further, when determining the first obstacle map according to the obstacle point cloud data, it includes: Project all the point cloud data points that generate the non-light markers onto the occupancy grid map, and based on the position information of each point cloud data point, determine whether there are point cloud data points with non-light markers in each grid cell of the occupancy grid map to determine the first obstacle map; When there are point cloud data points with non-light markers in the grid cell, mark the grid cell as occupied by an obstacle; When there are no point cloud data points with non-light markers in the grid cell, mark the grid cell as not occupied by an obstacle; Determine the first obstacle map based on the obstacle occupancy state and the non-obstacle occupancy state.

[0011] Further, when pruning the obstacle point cloud data according to the reflective area to obtain the light-avoiding point cloud data and determining the radar point cloud data according to the ultrasonic radar, it includes: Prune the point cloud data points that generate the reflective markers, and determine the remaining point cloud data points that generate the non-light markers as the light-avoiding point cloud data; Obtain the round-trip time of the echo of the ultrasonic radar, determine the distance information according to the round-trip time of the echo, convert the distance information into three-dimensional space coordinates, and construct a radar coordinate system; Convert each coordinate point of the radar coordinate system in the point cloud data format to determine radar data points, and determine all the radar data points as the radar point cloud data.

[0012] Further, when fusing the light-avoiding point cloud data and the radar point cloud data to determine the target obstacle point cloud data, it includes: Compare the light-avoiding point cloud data and the radar point cloud data for coincidence to determine the coincident points and non-coincident points with overlapping positions; Based on the DBSCAN algorithm, with each coincident point as the center, set a neighborhood radius, search for and extract all the clustering data points within this neighborhood, and the clustering data points include the radar data points and the point cloud data points that generate the non-light markers; Preset the minimum number of points, obtain the number of data points of the clustering data points within the neighborhood radius of each coincident point, and compare the number of data points with the minimum number of points to determine whether this coincident point is a core point.

[0013] Further, when comparing the number of data points with the minimum number of points to determine whether this coincident point is a core point, it includes: When the number of data points is greater than or equal to the minimum number of points, determine this coincident point as a core point, expand its neighborhood radius, and determine the clustering set; When the number of the data points is less than the minimum number of points, and the clustered data points belong to the neighborhood radius determined as the core points, the coincident points are determined as border points; When the number of the data points is less than the minimum number of points, and the clustered data points do not belong to the neighborhood radius determined as the core points, the coincident points are determined as noise points.

[0014] Further, the obstacle avoidance method based on multi-sensor fusion further includes: merging the clustering set, the border points and the non-coincident points to determine the target obstacle point cloud data.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: The target image to be avoided is determined according to several images to be avoided, and the obstacle point cloud data is generated accordingly, ensuring the comprehensiveness of obtaining environmental information. With the help of the sunlight segmentation model, the reflective mask image can be determined according to the target image to be avoided. The interference of the 3D camera in the sunlight reflection environment is effectively identified, avoiding the noise points generated by the reflection and affecting the stability of obstacle avoidance, and improving the reliability and adaptability of obstacle avoidance. The first obstacle map is generated according to the obstacle point cloud data, ensuring the timeliness of map drawing. In the complex situation of sunlight reflection, by deleting the obstacle point cloud data affected by the reflection, the light avoidance point cloud data is determined, and the radar point cloud data of the ultrasonic radar is fused to generate the second obstacle map, which not only overcomes the limitations of the 3D camera in the reflective environment, but also combines the stable ranging characteristics of the ultrasonic radar, realizing the complementary advantages of the two sensors, thereby providing a reliable basis for the decision-making of obstacle avoidance. The multi-sensor fusion method no longer relies on a single sensor, enhancing the stability and adaptability of the overall process of obstacle avoidance.

[0016] On the other hand, the present application also provides an obstacle avoidance system based on multi-sensor fusion, which is applied to the above-mentioned obstacle avoidance method based on multi-sensor fusion, and includes: An acquisition module, configured to acquire several images to be avoided, process the several images to be avoided to determine the target image to be avoided, and determine the obstacle point cloud data according to the target image to be avoided; A judgment module, configured to determine the reflective mask image based on the sunlight segmentation model according to the target image to be avoided, extract the pixel values of the reflective mask image, and judge whether there is a reflective area according to the pixel values and the obstacle point cloud data; A processing module, configured to, when there is no such reflective area, determine the first obstacle map according to the obstacle point cloud data, and when there is such reflective area, delete the obstacle point cloud data according to the reflective area to obtain the light avoidance point cloud data, determine the radar point cloud data according to the ultrasonic radar, fuse the light avoidance point cloud data and the radar point cloud data to determine the target obstacle point cloud data, and determine the second obstacle map according to the target obstacle point cloud data; The obstacle avoidance module is configured to determine an obstacle avoidance path based on the first obstacle map or the second obstacle map.

[0017] It can be understood that the above-mentioned obstacle avoidance method and system based on multi-sensor fusion have the same beneficial effects, which will not be elaborated here. Description of the Drawings

[0018] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of illustrating the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings: Figure 1 It is a flowchart of an obstacle avoidance method based on multi-sensor fusion provided by an embodiment of the present invention.

[0019] Figure 2 It is a functional block diagram of an obstacle avoidance system based on multi-sensor fusion provided by an embodiment of the present invention. Detailed Embodiments

[0020] Hereinafter, exemplary embodiments of the present disclosure will be described in more detail with reference to the drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be completely conveyed to those skilled in the art. It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other. Hereinafter, the present invention will be described in detail with reference to the drawings and in conjunction with the embodiments.

[0021] In some embodiments of the present application, referring to Figure 1 As shown, an obstacle avoidance method based on multi-sensor fusion includes: S100: Obtain a plurality of images to be avoided, process the plurality of images to be avoided, determine the target image to be avoided, and determine the obstacle point cloud data according to the target image to be avoided.

[0022] S200: Based on the target image to be avoided, determine a specular mask image based on a sunlight segmentation model, extract the pixel values of the specular mask image, and determine whether there is a specular area according to the pixel values and the obstacle point cloud data.

[0023] S300: When there is no reflective area, determine the first obstacle map based on the obstacle point cloud data. When there is a reflective area, delete the obstacle point cloud data according to the reflective area to obtain the light-avoiding point cloud data, and determine the radar point cloud data based on the ultrasonic radar. Integrate the light-avoiding point cloud data and the radar point cloud data to determine the target obstacle point cloud data, and determine the second obstacle map based on the target obstacle point cloud data.

[0024] S400: Determine the obstacle avoidance path based on the first obstacle map or the second obstacle map.

[0025] Specifically, during the movement of the robot, first obtain a number of obstacle avoidance images of the 3D camera (binocular 3D camera) deployed on the robot. The number of obstacle avoidance images is preferably 10, and can be adjusted according to the actual environment. These obstacle avoidance images cover the visual information of the environment around the robot. Since the obstacle avoidance images are taken from multiple angles (such as the left front, the front, and the right front, etc.), there will be a certain deviation between them. Process these obstacle avoidance images, and the processing methods include denoising and geometric correction, etc., so as to improve the accuracy of the obstacle avoidance target images. Use the obstacle avoidance target images to establish the obstacle point cloud data. Processing in the point cloud environment can ensure the consistency of the coordinate system and avoid deviations in subsequent judgments. The obstacle point cloud data intuitively reflects the position and approximate shape of the obstacles, providing basic data for subsequent analysis. There may be factors such as sunlight reflected by the ground in the obstacle avoidance target images, resulting in misjudgment or missed judgment of the obstacles. Therefore, use the sunlight segmentation model to determine the reflective mask image. The sunlight segmentation model analyzes the light characteristics in the obstacle avoidance target images to generate the reflective mask image, and extracts the pixel values of the reflective mask image to combine with the previously obtained obstacle point cloud data to judge whether there is a reflective area in the current environment. This ensures the stability of obstacle avoidance under complex lighting conditions. When there is no reflective area, the first obstacle map can be directly determined based on the obstacle point cloud data. In this case, the obstacle point cloud data can truly reflect the distribution of the obstacles, ensuring the reliability of the first obstacle map. When there is a reflective area, there will be error information such as noise in the obstacle point cloud data. Therefore, it is necessary to delete some data to obtain the light-avoiding point cloud data. At the same time, use the ultrasonic radar to obtain the radar point cloud data. The ultrasonic radar measures the distance to the obstacle by emitting and receiving ultrasonic waves, so as to obtain the radar point cloud data. The ultrasonic radar is not affected by sunlight reflection, but cannot obtain data such as the contour of the obstacle. By integrating the light-avoiding point cloud data determined by the 3D camera and the radar point cloud data, the advantages of the two devices can be fully utilized, complementing each other's deficiencies, and then determining the target obstacle point cloud data, and generating the second obstacle map based on this, ensuring that accurate obstacle information can still be obtained when there is interference from reflection, and improving the adaptability of obstacle avoidance in complex environments.

[0026] It can be understood that the A* algorithm, Dijkstra algorithm, etc. are used to determine the obstacle avoidance path according to the first obstacle map or the second obstacle map. Specifically, one of them can be selected according to the actual situation, which is not limited here. Compared with the obstacle avoidance of a single sensor, the fusion of multiple sensors can provide rich and accurate environmental information, making the planned obstacle avoidance path safe and reliable, and improving the stability of the robot's obstacle avoidance.

[0027] In some embodiments of the present application, when processing a plurality of images to be avoided and determining the target image to be avoided, it includes: preprocessing a plurality of images to be avoided, and the preprocessing includes denoising and geometric correction; extracting feature points from the preprocessed plurality of images to be avoided according to the Speeded Up Robust Features (SURF) algorithm; using the Random Sample Consensus (RANSAC) algorithm to match the extracted feature points, determining the image correlation data between the preprocessed plurality of images to be avoided; registering the preprocessed plurality of images to be avoided based on the image correlation data; and merging the registered plurality of images to be avoided according to multi-band fusion to determine the target image to be avoided.

[0028] Specifically, preprocessing a plurality of images to be avoided improves the image quality and lays a foundation for subsequent feature point extraction and registration. Using the SURF algorithm can determine the image correlation data between images, and the image correlation data includes relative position and rotation relationship. Based on affine transformation, perspective transformation, etc., the preprocessed plurality of images to be avoided are registered, eliminating factors such as translation and rotation caused by different shooting angles between images, achieving precise alignment between images. Using multi-band fusion, etc. to merge the registered multi-angle images ensures the overall visual effect and stitching quality of the target image to be avoided.

[0029] In some embodiments of the present application, when determining the reflective mask image based on the sunlight segmentation model according to the target image to be avoided, it includes: obtaining an image data set, dividing the image data set into a training set and a test set; using cross-validation and grid search to find the model parameters of the neural network model, establishing the neural network model; fitting the neural network model according to the training set, and adjusting the hyperparameters of the neural network model according to the root mean square error of the test set; substituting the test set into the adjusted neural network model and calculating the accuracy rate; when the accuracy rate reaches the preset accuracy threshold, determining the reflective mask image according to the target image to be avoided.

[0030] Specifically, the obtained image dataset includes key data such as sunlight area images during various sunlight reflections in the current environment, a three-dimensional coordinate system, point cloud environment data, etc. These data record the sunlight reflection conditions at different time periods. The image dataset is divided into a training set and a test set. Usually, 70% - 80% of the data is used as the training set, and the rest is used as the test set. Ensure that both the training set and the test set contain data on various sunlight reflection conditions to improve the generalization ability of the model. Cross-validation and Grid Search are used to find the model parameters of the neural network model. Cross-validation divides the data into several parts and trains the model multiple times to verify its stability and performance. Grid Search exhaustively searches parameter combinations in the parameter space to establish the neural network model. The neural network model contains multiple layers, different types of neurons, and activation functions, aiming to capture complex relationships in the data. When the test set is substituted into the neural network model, the root mean square error of the test set will be obtained, and based on this, the hyperparameters of the neural network model are adjusted to optimize the generalization ability of the model, enabling it to maintain good prediction performance on unseen data. Then, the test set is substituted into the adjusted neural network model to calculate the accuracy rate. The accuracy rate is an important indicator for evaluating the model performance. After the model reaches the preset accuracy threshold, it indicates that the model performance has remained stable, and it is considered that the model has reached a satisfactory performance level and can determine the reflective mask image based on the image of the obstacle to be avoided.

[0031] In some embodiments of the present application, when extracting the pixel values of the reflective mask image and determining whether there is a reflective area based on the pixel values and the obstacle point cloud data, it includes: extracting all the point cloud data points corresponding in the obstacle point cloud data, extracting all the mask pixel points corresponding to the reflective mask image, and determining the pixel values of all the mask pixel points. When the pixel value of the mask pixel point corresponding to the point cloud data point is 1, then generate a reflective mark for this point cloud data point. When the pixel value of the mask pixel point corresponding to the point cloud data point is 0, then generate a non-reflective mark for this point cloud data point. If a reflective mark is recognized, it is determined that there is a reflective area. If no reflective mark is recognized, it is determined that there is no reflective area.

[0032] Specifically, since the reflective mask image is determined based on the image of the obstacle to be avoided and the registration operation has been performed when determining the image of the obstacle to be avoided, therefore, the reflective mask image and the image of the obstacle to be avoided maintain the coordinate system of the same point cloud environment, and moreover, the point cloud data points corresponding in the obstacle point cloud data and the mask pixel points corresponding to the reflective mask image are in one-to-one correspondence. The reflective mask image is a binary image, and the pixel values of the mask pixel points in it are only 0 or 1. 0 represents non-reflective pixel points, and 1 represents reflective pixel points. By finding the pixel value of the mask pixel point at the corresponding position through the point cloud data point, it can dynamically determine whether there is interference from sunlight reflection in the point cloud data point, ensuring the stability and accuracy of obstacle avoidance.

[0033] In some embodiments of the present application, when determining the first obstacle map based on the obstacle point cloud data, it includes: projecting all the point cloud data points generating non-light marks onto the occupancy grid map, and based on the position information of each point cloud data point, determining whether there are point cloud data points with non-light marks in each grid cell of the occupancy grid map to determine the first obstacle map. When there are point cloud data points with non-light marks in the grid cell, the grid cell is marked as occupied by an obstacle. When there are no point cloud data points with non-light marks in the grid cell, the grid cell is marked as not occupied by an obstacle. The first obstacle map is determined based on the occupied and non-occupied states of the obstacles.

[0034] Specifically, the occupancy grid map divides the space of the current environment into several grid cells, preferably 30 grid cells. After the point cloud data points with non-light marks are projected onto the occupancy grid map, the spatial position of each point cloud data point will correspond to one of the grid cells. By determining whether there are point cloud data points with non-light marks in each grid cell, the state of the grid cell can be determined. When there are point cloud data points with non-light marks in the grid cell, it indicates that there is an obstacle at this position, and the grid cell is marked as occupied by an obstacle. Otherwise, it is marked as not occupied by an obstacle. Based on the states of all grid cells, the first obstacle map reflecting the distribution of obstacles in the environment can be constructed, which is applicable to various complex environments. Regardless of the shape, size, and distribution of the obstacles, it can effectively identify and mark them, improving the stability and flexibility of obstacle avoidance.

[0035] In some embodiments of the present application, when deleting the obstacle point cloud data according to the reflective area to obtain the light-avoiding point cloud data and determining the radar point cloud data according to the ultrasonic radar, it includes: deleting the point cloud data points generating reflective marks, and determining the remaining point cloud data points generating non-light marks as the light-avoiding point cloud data, obtaining the round-trip time of the echo of the ultrasonic radar, determining the distance information according to the round-trip time, converting the distance information into three-dimensional space coordinates, constructing a radar coordinate system, converting each coordinate point of the radar coordinate system in the point cloud data format to determine the radar data points, and determining all the radar data points as the radar point cloud data.

[0036] Specifically, in the current environment, the reflective area interferes with the accuracy of the obstacle point cloud data. By deleting the point cloud data points that generate the reflective markers, the interference of reflection is eliminated, and the remaining point cloud data points without light markers are determined as the light-avoiding point cloud data, improving the accuracy of the light-avoiding point cloud data in reflecting the current environmental conditions, avoiding the noise generated by the reflective area from interfering with subsequent judgments, and thus improving the accuracy of obstacle avoidance. The ultrasonic radar detects by emitting ultrasonic waves and receiving the echoes reflected from obstacles. There is a fixed mathematical relationship between the round-trip time of the echoes and the distance, and based on this, the distance information of the obstacles can be calculated. The distance information is converted into three-dimensional space coordinates. At this time, the coordinate system corresponding to the three-dimensional space coordinates is translated or matrix-transformed so that the constructed radar coordinate system is consistent with the coordinate system of the point cloud environment. Then, each coordinate point is transformed according to the point cloud data format to form radar data points, and finally, the radar point cloud data is obtained. The techniques of translation or matrix transformation and the transformation of coordinate points into the point cloud data format are cumbersome and mature, and will not be introduced in detail here. By removing the point cloud data points in the reflective area, the interference of factors such as sunlight reflected from the ground on the data is effectively avoided. Moreover, the ultrasonic radar makes up for the removal of the point cloud data points of the reflective markers. By combining the light-avoiding point cloud data and the radar point cloud data, the advantages of different sensors are fully utilized, the comprehensiveness of environmental information is improved, the deficiencies of a single sensor are made up, the reliability and stability of obstacle avoidance are enhanced, and the risk of obstacle avoidance failure caused by data errors is reduced.

[0037] In some embodiments of the present application, when fusing the light-avoiding point cloud data and the radar point cloud data to determine the target obstacle point cloud data, it includes: comparing the light-avoiding point cloud data and the radar point cloud data for coincidence to determine the coincident points and non-coincident points with overlapping positions. Based on the DBSCAN algorithm, with each coincident point as the center, a neighborhood radius is set, and all the clustering data points within this neighborhood are searched and extracted. The clustering data points include radar data points and the point cloud data points that generate the lightless markers. A minimum number of points is preset, the number of data points of the clustering data points within the neighborhood radius of each coincident point is obtained, and the number of data points is compared with the minimum number of points to determine whether this coincident point is a core point.

[0038] Specifically, since the constructed radar coordinate system is consistent with the coordinate system of the point cloud environment, the point cloud data points contained in the light-avoiding point cloud data are consistent with the positions of the radar data points contained in the radar point cloud data. By overlapping and comparing the light-avoiding point cloud data and the radar point cloud data, the overlapping points and non-overlapping points with overlapping positions can be distinguished. The overlapping points indicate that different sensors detect the same obstacle at this position. Since the ultrasonic radar is not affected by sunlight reflection, the non-overlapping points reflect the point cloud data points for deleting and generating reflective marks, and the corresponding supplement made by the ultrasonic radar for this deleted part. During the fusion process, in-depth analysis of the overlapping points is carried out to avoid redundancy or errors caused by simple data superposition, so that the fused data can accurately reflect the actual distribution of obstacles. Based on the density characteristics of the DBSCAN algorithm, the noise points in the low-density area can be effectively identified and filtered out, improving the quality of the target obstacle point cloud data, enhancing the adaptability and reliability of obstacle avoidance, and providing reliable data support for subsequent obstacle avoidance path planning.

[0039] In some embodiments of the present application, when comparing the number of data points with the minimum number of points to determine whether the overlapping point is a core point, it includes: when the number of data points is greater than or equal to the minimum number of points, determining the overlapping point as a core point, expanding its neighborhood radius, and determining the clustering set; when the number of data points is less than the minimum number of points and the clustering data points belong to the neighborhood radius determined as the core point, determining the overlapping point as a boundary point; when the number of data points is less than the minimum number of points and the clustering data points do not belong to the neighborhood radius determined as the core point, determining the overlapping point as a noise point.

[0040] In some embodiments of the present application, the obstacle avoidance method for multi-sensor fusion further includes: merging the clustering set, boundary points, and non-overlapping points to determine the target obstacle point cloud data.

[0041] Specifically, based on the principle of DBSCAN algorithm, when the number of data points in the neighborhood of a certain coincident point is greater than or equal to the minimum number of points, it means that the point cloud distribution in the neighborhood is relatively dense, and it is determined as a core point. By expanding the neighborhood radius of the core point, more adjacent cluster data points can be absorbed to determine the cluster set, and then determine the main contour and other features of the obstacle. When the number of data points is less than the minimum number of points, but the cluster data points are within the neighborhood radius of the determined core point, although these cluster data points have insufficient neighborhood density, they are related to the neighborhood where the core point is located, so they are determined as boundary points, representing the edge of the obstacle. And those cluster data points whose number of data points is less than the minimum number of points and are not within the neighborhood radius of the core point are judged as noise points because they are in a low-density isolated area. The cluster set, boundary points and non-coincidence points are merged. The cluster set reflects the main structure of the obstacle, the boundary points supplement the edge information of the obstacle, and the non-coincidence points cover the supplementary data of the ultrasonic sensor. The three are merged to fully integrate the multi-sensor data, so as to determine the target obstacle point cloud data. Through reasonable classification and merging, the integrity of the target obstacle point cloud data is ensured, so as to accurately depict the overall picture of the obstacle and avoid missing information. In addition, noise points are identified and not merged, which effectively reduces the impact of environmental interference and sensor errors and improves the reliability of obstacle avoidance.

[0042] It can be understood that the method of determining the second obstacle map based on the target obstacle point cloud data is consistent with the method of determining the first obstacle map based on the obstacle point cloud data, except that the point cloud data points are different. Therefore, the description of the method of determining the second obstacle map is redundant and will not be repeated here.

[0043] In summary, the beneficial effects of the present invention are: determining the target image to be avoided based on a number of images to be avoided, and generating obstacle point cloud data accordingly, thereby ensuring the comprehensiveness of environmental information. With the help of the sunlight segmentation model, the reflective mask image can be determined based on the target image to be avoided. The interference of the 3D camera in the sunlight reflection environment is effectively identified, and the noise generated by the reflection is avoided to affect the stability of obstacle avoidance, thereby improving the reliability and adaptability of obstacle avoidance. The first obstacle map is generated according to the obstacle point cloud data, ensuring the timeliness of map drawing. In the complex case of sunlight reflection, by deleting the obstacle point cloud data affected by the reflection, determining the light avoidance point cloud data, and fusing the radar point cloud data of the ultrasonic radar, the second obstacle map is generated, which not only overcomes the limitations of the 3D camera in the reflective environment, but also combines the stable ranging characteristics of the ultrasonic radar, realizes the complementary advantages of the two sensors, and thus provides a reliable basis for the decision-making of obstacle avoidance. The multi-sensor fusion method no longer relies on a single sensor, which enhances the stability and adaptability of the overall obstacle avoidance process.

[0044] In another preferred embodiment based on the above embodiment, refer toFigure 2 As shown in Figure 2 , this embodiment provides an obstacle avoidance system for multi-sensor fusion, which is applied to the above-mentioned multi-sensor fusion obstacle avoidance method, and includes: An acquisition module, configured to obtain a plurality of images to be avoided obstacles, process the plurality of images to be avoided obstacles, determine an image of the target to be avoided obstacles, and determine obstacle point cloud data according to the image of the target to be avoided obstacles; A judgment module, configured to determine a reflective mask image based on a sunlight segmentation model according to the image of the target to be avoided obstacles, extract pixel values of the reflective mask image, and judge whether there is a reflective area according to the pixel values and the obstacle point cloud data; A processing module, configured to, when there is no reflective area, determine a first obstacle map according to the obstacle point cloud data, and when there is a reflective area, delete the obstacle point cloud data according to the reflective area to obtain avoidance point cloud data, determine radar point cloud data according to an ultrasonic radar, fuse the avoidance point cloud data and the radar point cloud data to determine target obstacle point cloud data, and determine a second obstacle map according to the target obstacle point cloud data; An obstacle avoidance module, configured to determine an obstacle avoidance path based on the first obstacle map or the second obstacle map.

[0045] Specifically, determining the image of the target to be avoided obstacles according to a plurality of images to be avoided obstacles and generating obstacle point cloud data accordingly ensures the comprehensiveness of obtaining environmental information. With the help of the sunlight segmentation model, a reflective mask image can be determined according to the image of the target to be avoided obstacles. It effectively identifies the interference of the 3D camera in the sunlight reflection environment, avoids the noise generated by the reflection light from affecting the stability of obstacle avoidance, and improves the reliability and adaptability of obstacle avoidance. Generating the first obstacle map according to the obstacle point cloud data ensures the timeliness of map drawing. In the complex situation of sunlight reflection, by deleting the obstacle point cloud data affected by the reflection light to determine the avoidance point cloud data, and fusing the radar point cloud data of the ultrasonic radar to generate the second obstacle map, it not only overcomes the limitations of the 3D camera in the reflection environment, but also combines the stable ranging characteristics of the ultrasonic radar, realizes the complementary advantages of the two sensors, and thus provides a reliable basis for the decision-making of obstacle avoidance. The multi-sensor fusion method no longer relies on a single sensor, enhancing the stability and adaptability of the overall obstacle avoidance process.

[0046] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0047] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, as well as the combination of flows and / or blocks in the flowchart and / or block diagram. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device generate means for implementing the functions specified in the Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0048] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in the Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0049] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operating steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in the Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: it is still possible to modify the specific embodiments of the present invention or make equivalent substitutions, and any modification or equivalent substitution that does not depart from the spirit and scope of the present invention should be covered by the protection scope of the claims of the present invention.

Claims

1. A multi-sensor fusion obstacle avoidance method, characterized in that: include: Obtaining a plurality of images of obstacles to be avoided, and processing the plurality of images of obstacles to be avoided to determine target images of obstacles to be avoided, and determining obstacle point cloud data according to the target images of obstacles to be avoided; According to the obstacle avoidance target image, a reflective mask image is determined based on a sunlight segmentation model, pixel values ​​of the reflective mask image are extracted, and whether there is a reflective area is determined according to the pixel values ​​and the obstacle point cloud data; If the reflective area does not exist, a first obstacle map is determined according to the obstacle point cloud data; if the reflective area exists, the obstacle point cloud data is deleted according to the reflective area to obtain light-avoiding point cloud data, and radar point cloud data is determined according to the ultrasonic radar, the light-avoiding point cloud data and the radar point cloud data are fused to determine target obstacle point cloud data, and a second obstacle map is determined according to the target obstacle point cloud data; An obstacle avoidance path is determined based on the first obstacle map or the second obstacle map.

2. The obstacle avoidance method of multi-sensor fusion according to claim 1, characterized in that: When processing a plurality of obstacle avoidance images to determine an obstacle avoidance target image, the process includes: Preprocessing a plurality of the obstacle avoidance images, wherein the preprocessing includes denoising and geometric correction; The feature points of several pre-processed images to be avoided are extracted according to the accelerated robust feature algorithm, and the extracted feature points are matched using the RANSAC algorithm to determine the image correlation data between the pre-processed images to be avoided; Based on the image correlation data, a plurality of pre-processed images to be avoided by obstacles are registered, and the registered images to be avoided by obstacles are merged according to multi-band fusion to determine the target image to be avoided by obstacles.

3. The obstacle avoidance method of multi-sensor fusion according to claim 2, characterized in that: When determining the reflective mask image based on the sunlight segmentation model according to the obstacle avoidance target image, the method includes: Acquire an image data set, divide the image data set into a training set and a test set, use cross-validation and grid search to find model parameters of a neural network model, and establish a neural network model; Fitting the neural network model according to the training set, and adjusting the hyperparameters of the neural network model according to the root mean square error of the test set; The test set is substituted into the adjusted neural network model and the accuracy is calculated. When the accuracy reaches a preset accuracy threshold, the reflective mask image is determined according to the obstacle avoidance target image.

4. The obstacle avoidance method of multi-sensor fusion according to claim 3, characterized in that: When extracting the pixel value of the reflective mask image and judging whether there is a reflective area according to the pixel value and the obstacle point cloud data, the method includes: Extracting all corresponding point cloud data points in the obstacle point cloud data, extracting all mask pixel points corresponding to the reflective mask image, and determining pixel values ​​of all mask pixel points; When the pixel value of the mask pixel corresponding to the point cloud data point is 1, a reflective mark is generated for the point cloud data point; When the pixel value of the mask pixel corresponding to the point cloud data point is 0, a matte mark is generated for the point cloud data point; If the reflective mark is identified, it is determined that there is a reflective area; If the reflective mark is not identified, it is determined that there is no reflective area.

5. The obstacle avoidance method of multi-sensor fusion according to claim 4, characterized in that: When determining a first obstacle map according to the obstacle point cloud data, the method includes: Projecting all point cloud data points with no light mark generated onto an occupancy grid map, and judging whether there is a point cloud data point with no light mark in each grid unit of the occupancy grid map according to the position information of each point cloud data point, so as to determine the first obstacle map; When there is a point cloud data point without light mark in a grid cell, the grid cell is marked as being in an obstacle occupied state; When there is no point cloud data point without light mark in the grid cell, the grid cell is marked as a non-obstacle occupied state; The first obstacle map is determined according to the obstacle occupancy state and the non-obstacle occupancy state.

6. The obstacle avoidance method of multi-sensor fusion according to claim 5, characterized in that: When the obstacle point cloud data is deleted according to the reflective area to obtain the light-shielding point cloud data, and the radar point cloud data is determined according to the ultrasonic radar, the method includes: Deleting the point cloud data points for generating the reflective mark, and determining the remaining point cloud data points for generating the matte mark as the light-shielding point cloud data; Obtaining the round-trip time of the echo of the ultrasonic radar, determining distance information according to the round-trip time of the echo, converting the distance information into three-dimensional space coordinates, and constructing a radar coordinate system; Each coordinate point of the radar coordinate system is converted into a point cloud data format to determine a radar data point, and all radar data points are determined as the radar point cloud data.

7. The obstacle avoidance method of multi-sensor fusion according to claim 6, characterized in that: When the light-shielding point cloud data and the radar point cloud data are merged to determine the target obstacle point cloud data, the method includes: Comparing the light-shielding point cloud data with the radar point cloud data to determine the overlapping points and non-overlapping points; Based on the DBSCAN algorithm, with each coincident point as the center, a neighborhood radius is set, and all clustered data points in the neighborhood are searched and extracted, wherein the clustered data points include the radar data points and the point cloud data points for generating the light-free mark; The minimum number of points is preset, the number of data points of clustered data points within the neighborhood radius of each coincident point is obtained, and the number of data points is compared with the minimum number of points to determine whether the coincident point is a core point.

8. The obstacle avoidance method of multi-sensor fusion according to claim 7, characterized in that: When comparing the number of data points with the minimum number of points to determine whether the coincident point is a core point, the method includes: When the number of data points is greater than or equal to the minimum number of points, the coincident point is determined as a core point, its neighborhood radius is expanded, and a cluster set is determined; When the number of data points is less than the minimum number of points, and the clustered data points belong to the neighborhood radius determined as the core point, the coincident point is determined as a boundary point; When the number of data points is less than the minimum number of points, and the clustered data point does not belong to the neighborhood radius determined as the core point, the coincident point is determined as a noise point.

9. The obstacle avoidance method of multi-sensor fusion according to claim 8, characterized in that: Also includes: The cluster set, the boundary points and the non-overlapping points are merged to determine the target obstacle point cloud data.

10. A multi-sensor fusion obstacle avoidance system, applied to the multi-sensor fusion obstacle avoidance method according to any one of claims 1 to 9, characterized in that: include: The acquisition module is configured to acquire a plurality of images of obstacles to be avoided, process the plurality of images of obstacles to be avoided, determine the target images of obstacles to be avoided, and determine the obstacle point cloud data according to the target images of obstacles to be avoided; A judgment module is configured to determine a reflective mask image based on the sunlight segmentation model according to the obstacle avoidance target image, extract pixel values ​​of the reflective mask image, and judge whether there is a reflective area according to the pixel values ​​and the obstacle point cloud data; The processing module is configured to determine a first obstacle map according to the obstacle point cloud data if the reflective area does not exist, and to delete the obstacle point cloud data according to the reflective area to obtain light-avoiding point cloud data if the reflective area exists, and to determine radar point cloud data according to the ultrasonic radar, to fuse the light-avoiding point cloud data with the radar point cloud data to determine target obstacle point cloud data, and to determine a second obstacle map according to the target obstacle point cloud data; The obstacle avoidance module is configured to determine an obstacle avoidance path based on the first obstacle map or the second obstacle map.

Images