An Obstacle Avoidance Method and System Based on Multi-Sensor Fusion

Abstract

The present invention relates to the technical field of obstacle avoidance, and discloses an obstacle avoidance method and system based on multi-sensor fusion. The method includes: determining an image of an obstacle to be avoided, and determining obstacle point cloud data according to the image of the obstacle to be avoided. Determining a specular reflection mask image based on a sunlight segmentation model, extracting pixel values of the specular reflection mask image, and judging whether there is a specular reflection area according to the pixel values and the obstacle point cloud data. When there is no specular reflection area, determining a first obstacle map according to the obstacle point cloud data; when there is a specular reflection area, deleting the obstacle point cloud data according to the specular reflection area, determining radar point cloud data according to an ultrasonic radar, fusing the specular reflection avoidance point cloud data and the radar point cloud data, and determining an obstacle avoidance path based on the first obstacle map or a second obstacle map. By identifying the specular reflection area and fusing the specular reflection avoidance point cloud data and the radar point cloud data, the present invention improves the reliability of obstacle avoidance.

Description

Technical Field

[0001] The present invention relates to the technical field of obstacle avoidance, and more particularly, to an obstacle avoidance method and system based on multi-sensor fusion. Background Art

[0002] With the booming development of the robot industry, mobile robots have been widely popularized in many fields, and higher requirements have been put forward for the safety, obstacle detection, obstacle avoidance, and path planning of robots. Especially in complex scenarios, there are certain challenges in aspects such as obstacle detection and smooth movement.

[0003] When a robot performs obstacle detection, in order to accurately avoid obstacles, it is necessary to make a judgment based on the three-dimensional information of the three-dimensional environment around the robot. Currently, robots usually use a 3D camera (such as a binocular 3D camera) to detect obstacles through point cloud data. However, when there is sunlight reflection on the ground, noise points will be generated in the reflective area of the point cloud data, resulting in false detection or missed detection of obstacles. A single sensor cannot meet the detection requirements under complex lighting conditions.

[0004] Therefore, it is necessary to design an obstacle avoidance method and system based on multi-sensor fusion to solve the problems existing in the current technology. Summary of the Invention

[0005] In view of this, the present invention proposes an obstacle avoidance method and system based on multi-sensor fusion, aiming to solve the problem that when there is sunlight reflection on the ground, noise points will be generated in the reflective area of the point cloud data, resulting in false detection or missed detection of obstacles, and a single sensor cannot meet the detection requirements under complex lighting conditions.

[0006] On the one hand, the present invention proposes an obstacle avoidance method based on multi-sensor fusion, including:

[0007] Obtaining a plurality of images to be avoided, processing the plurality of images to be avoided to determine a target image to be avoided, and determining obstacle point cloud data according to the target image to be avoided;

[0008] Based on the target image to be avoided, determining a reflective mask image based on a sunlight segmentation model, extracting pixel values of the reflective mask image, and judging whether there is a reflective area according to the pixel values and the obstacle point cloud data;

[0009] When there is no such reflective area, determining a first obstacle map according to the obstacle point cloud data; when there is such a reflective area, deleting the obstacle point cloud data according to the reflective area to obtain light-avoiding point cloud data, determining radar point cloud data according to an ultrasonic radar, fusing the light-avoiding point cloud data and the radar point cloud data to determine target obstacle point cloud data, and determining a second obstacle map according to the target obstacle point cloud data;

[0010] Determine an obstacle avoidance path based on the first obstacle map or the second obstacle map.

[0011] Furthermore, when processing a plurality of the images to be avoided and determining the target image to be avoided, it includes:

[0012] Preprocess a plurality of the images to be avoided, and the preprocessing includes denoising and geometric correction;

[0013] Extract feature points from the preprocessed plurality of images to be avoided according to the Speeded-Up Robust Features (SURF) algorithm, match the extracted feature points using the Random Sample Consensus (RANSAC) algorithm, and determine the image correlation data between the preprocessed plurality of images to be avoided;

[0014] Register the preprocessed plurality of images to be avoided based on the image correlation data, and merge the registered plurality of images to be avoided according to multi-band fusion to determine the target image to be avoided.

[0015] Furthermore, when determining a specular reflection mask image based on the target image to be avoided and a sunlight segmentation model, it includes:

[0016] Obtain an image dataset, divide the image dataset into a training set and a test set, use cross-validation and grid search to find the model parameters of a neural network model, and establish a neural network model;

[0017] Fit the neural network model according to the training set, and adjust the hyperparameters of the neural network model according to the root mean square error of the test set;

[0018] Substitute the test set into the adjusted neural network model and calculate the accuracy rate. When the accuracy rate reaches a preset accuracy rate threshold, determine the specular reflection mask image according to the target image to be avoided.

[0019] Furthermore, when extracting the pixel values of the specular reflection mask image and determining whether there is a specular reflection area according to the pixel values and the obstacle point cloud data, it includes:

[0020] Extract all the point cloud data points corresponding in the obstacle point cloud data, extract all the mask pixel points corresponding to the specular reflection mask image, and determine the pixel values of all the mask pixel points;

[0021] When the pixel value of the mask pixel point corresponding to the point cloud data point is 1, generate a specular reflection mark for the point cloud data point;

[0022] When the pixel value of the mask pixel point corresponding to the point cloud data point is 0, generate a non-specular reflection mark for the point cloud data point;

[0023] If the reflective marker is recognized, it is determined that there is a reflective area;

[0024] If the reflective marker is not recognized, it is determined that there is no reflective area.

[0025] Further, when determining the first obstacle map based on the obstacle point cloud data, it includes:

[0026] Project all the point cloud data points generating non-light markers onto the occupancy grid map, and based on the position information of each point cloud data point, determine whether there are point cloud data points of non-light markers in each grid cell of the occupancy grid map to determine the first obstacle map;

[0027] When there are point cloud data points of non-light markers in the grid cell, mark this grid cell as the occupied state by an obstacle;

[0028] When there are no point cloud data points of non-light markers in the grid cell, mark this grid cell as the non-occupied state by an obstacle;

[0029] Determine the first obstacle map based on the occupied state by an obstacle and the non-occupied state by an obstacle.

[0030] Further, when deleting the obstacle point cloud data according to the reflective area to obtain the light-avoiding point cloud data and determining the radar point cloud data according to the ultrasonic radar, it includes:

[0031] Delete the point cloud data points generating the reflective marker, and determine the remaining point cloud data points generating the non-light marker as the light-avoiding point cloud data;

[0032] Obtain the round-trip time of the echo of the ultrasonic radar, determine the distance information according to the round-trip time of the echo, convert the distance information into three-dimensional space coordinates, and construct a radar coordinate system;

[0033] Convert each coordinate point of the radar coordinate system in the point cloud data format to determine radar data points, and determine all the radar data points as the radar point cloud data.

[0034] Further, when fusing the light-avoiding point cloud data and the radar point cloud data to determine the target obstacle point cloud data, it includes:

[0035] Perform a coincidence comparison between the light-avoiding point cloud data and the radar point cloud data to determine the coincident points and non-coincident points with coincident positions;

[0036] Based on the DBSCAN algorithm, with each coincident point as the center, set a neighborhood radius, search for and extract all the clustering data points within this neighborhood, and the clustering data points include the radar data points and the point cloud data points generating the non-light marker;

[0037] A minimum number of points is preset, the number of data points within the neighborhood radius of each coincidence point is obtained, and the number of data points is compared with the minimum number of points to determine whether the coincidence point is a core point.

[0038] Further, when comparing the number of data points with the minimum number of points to determine whether the coincidence point is a core point, it includes:

[0039] When the number of data points is greater than or equal to the minimum number of points, the coincidence point is determined as a core point, its neighborhood radius is expanded, and a clustering set is determined;

[0040] When the number of data points is less than the minimum number of points and the clustering data points belong to the neighborhood radius of the determined core point, the coincidence point is determined as a boundary point;

[0041] When the number of data points is less than the minimum number of points and the clustering data points do not belong to the neighborhood radius of the determined core point, the coincidence point is determined as a noise point.

[0042] Further, the multi-sensor fusion obstacle avoidance method further includes: merging the clustering set, the boundary points and the non-coincidence points to determine the target obstacle point cloud data.

[0043] Compared with the prior art, the beneficial effects of the present invention are as follows: The target image to be avoided is determined according to several images to be avoided, and the obstacle point cloud data is generated accordingly, ensuring the comprehensiveness of obtaining environmental information. With the help of the sunlight segmentation model, the reflective mask image can be determined according to the target image to be avoided. The interference of the 3D camera in the sunlight reflection environment is effectively identified, avoiding the noise points generated by the reflection and affecting the stability of obstacle avoidance, and improving the reliability and adaptability of obstacle avoidance. The first obstacle map is generated according to the obstacle point cloud data, ensuring the timeliness of map drawing. In the complex situation of sunlight reflection, by deleting the obstacle point cloud data affected by the reflection, the light avoidance point cloud data is determined, and the radar point cloud data of the ultrasonic radar is fused to generate the second obstacle map, which not only overcomes the limitations of the 3D camera in the reflective environment, but also combines the stable ranging characteristics of the ultrasonic radar, realizing the complementary advantages of the two sensors, thereby providing a reliable basis for the decision-making of obstacle avoidance. The multi-sensor fusion method no longer relies on a single sensor, enhancing the stability and adaptability of the overall obstacle avoidance process.

[0044] On the other hand, the present application also provides a multi-sensor fusion obstacle avoidance system, which is applied to the above multi-sensor fusion obstacle avoidance method, including:

[0045] The acquisition module is configured to obtain a plurality of obstacle avoidance images to be processed, process the plurality of obstacle avoidance images to be processed to determine an obstacle avoidance target image, and determine obstacle point cloud data according to the obstacle avoidance target image;

[0046] The judgment module is configured to determine a reflective mask image based on a sunlight segmentation model according to the obstacle avoidance target image, extract pixel values of the reflective mask image, and judge whether there is a reflective area according to the pixel values and the obstacle point cloud data;

[0047] The processing module is configured to, when there is no such reflective area, determine a first obstacle map according to the obstacle point cloud data, and when there is such reflective area, delete the obstacle point cloud data according to the reflective area to obtain light avoidance point cloud data, determine radar point cloud data according to an ultrasonic radar, fuse the light avoidance point cloud data and the radar point cloud data to determine target obstacle point cloud data, and determine a second obstacle map according to the target obstacle point cloud data;

[0048] The obstacle avoidance module is configured to determine an obstacle avoidance path based on the first obstacle map or the second obstacle map.

[0049] It can be understood that the above-mentioned obstacle avoidance method and system based on multi-sensor fusion have the same beneficial effects and will not be elaborated here. Description of the Drawings

[0050] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0051] Figure 1 It is a flowchart of an obstacle avoidance method based on multi-sensor fusion provided by an embodiment of the present invention.

[0052] Figure 2 It is a functional block diagram of an obstacle avoidance system based on multi-sensor fusion provided by an embodiment of the present invention. Detailed Embodiments

[0053] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art. It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the drawings and in combination with the embodiments.

[0054] In some embodiments of the present application, referring to Figure 1 as shown, a multi-sensor fusion obstacle avoidance method includes:

[0055] S100: Obtain a plurality of images to be obstacle-avoided, process the plurality of images to be obstacle-avoided to determine the target image to be obstacle-avoided, and determine the obstacle point cloud data according to the target image to be obstacle-avoided.

[0056] S200: Based on the target image to be obstacle-avoided, determine a specular reflection mask image based on the sunlight segmentation model, extract the pixel values of the specular reflection mask image, and determine whether there is a specular reflection area according to the pixel values and the obstacle point cloud data.

[0057] S300: If there is no specular reflection area, determine the first obstacle map according to the obstacle point cloud data. If there is a specular reflection area, delete the obstacle point cloud data according to the specular reflection area to obtain the specular reflection avoidance point cloud data, determine the radar point cloud data according to the ultrasonic radar, fuse the specular reflection avoidance point cloud data and the radar point cloud data to determine the target obstacle point cloud data, and determine the second obstacle map according to the target obstacle point cloud data.

[0058] S400: Determine the obstacle avoidance path based on the first obstacle map or the second obstacle map.

[0059] Specifically, during the movement of the robot, first, a number of obstacle avoidance images of the 3D camera (binocular 3D camera) deployed on the robot are obtained. The number of obstacle avoidance images is preferably 10, which can be adjusted according to the actual environment. These obstacle avoidance images cover the visual information of the robot's surrounding environment. Since the obstacle avoidance images are taken from multiple angles (such as the front left, directly in front, and front right), there will be a certain deviation between them. These obstacle avoidance images are processed, and the processing methods include denoising and geometric correction, etc., so as to improve the accuracy of the obstacle avoidance target images. The obstacle point cloud data is established using the obstacle avoidance target images and processed in the point cloud environment, which can ensure the consistency of the coordinate system and avoid the deviation of subsequent judgments. The obstacle point cloud data intuitively reflects the position and approximate shape of the obstacles, providing basic data for subsequent analysis. There may be factors such as sunlight reflected on the ground in the obstacle avoidance target images, resulting in misjudgment or missed judgment of the obstacles. Therefore, a sunlight segmentation model is used to determine the reflective mask image. The sunlight segmentation model analyzes the light characteristics in the obstacle avoidance target images, generates the reflective mask image, and extracts the pixel values of the reflective mask image to combine with the previously obtained obstacle point cloud data to judge whether there is a reflective area in the current environment. This ensures the stability of obstacle avoidance under complex lighting conditions. When there is no reflective area, the first obstacle map can be directly determined according to the obstacle point cloud data. In this case, the obstacle point cloud data can truly reflect the distribution of the obstacles, ensuring the reliability of the first obstacle map. When there is a reflective area, there will be error information such as noise in the obstacle point cloud data. Therefore, a part of the data needs to be deleted to obtain the light avoidance point cloud data. At the same time, the ultrasonic radar is used to obtain the radar point cloud data. The ultrasonic radar measures the distance to the obstacle by emitting and receiving ultrasonic waves, thereby obtaining the radar point cloud data. The ultrasonic radar is not affected by sunlight reflection, but it cannot obtain data such as the contour of the obstacle. By fusing the light avoidance point cloud data determined by the 3D camera and the radar point cloud data, the advantages of the two devices can be fully utilized, complementing each other's deficiencies, and then determining the target obstacle point cloud data, and generating the second obstacle map based on this, ensuring that accurate obstacle information can still be obtained when there is interference from reflection, and improving the obstacle avoidance adaptability in complex environments.

[0060] It can be understood that the A* algorithm, Dijkstra algorithm, etc. are used to determine the obstacle avoidance path according to the first obstacle map or the second obstacle map. Specifically, one of them can be selected according to the actual situation, which is not limited here. Compared with the obstacle avoidance of a single sensor, the fusion of multiple sensors can provide rich and accurate environmental information, making the planned obstacle avoidance path safe and reliable, and improving the stability of the robot's obstacle avoidance.

[0061] In some embodiments of the present application, when processing a number of obstacle avoidance images to determine an obstacle avoidance target image, it includes: preprocessing the number of obstacle avoidance images, the preprocessing including denoising and geometric correction, extracting feature points from the preprocessed number of obstacle avoidance images according to the Speeded Up Robust Features (SURF) algorithm, using the Random Sample Consensus (RANSAC) algorithm to match the extracted feature points, determining the image correlation data between the preprocessed number of obstacle avoidance images, registering the preprocessed number of obstacle avoidance images based on the image correlation data, and merging the registered number of obstacle avoidance images according to multi-band fusion to determine the obstacle avoidance target image.

[0062] Specifically, preprocessing the number of obstacle avoidance images improves the image quality and lays a foundation for subsequent feature point extraction and registration. Using the Speeded Up Robust Features (SURF) algorithm can determine the image correlation data between images, and the image correlation data includes relative position and rotation relationship. Based on affine transformation, perspective transformation, etc., the preprocessed number of obstacle avoidance images are registered to eliminate factors such as translation and rotation caused by different shooting angles between images, achieving precise alignment between images. Using multi-band fusion, etc., the registered multi-angle images are merged to ensure the overall visual effect and stitching quality of the obstacle avoidance target image.

[0063] In some embodiments of the present application, when determining a specular mask image based on a specular segmentation model according to the obstacle avoidance target image, it includes: obtaining an image data set, dividing the image data set into a training set and a test set, using cross-validation and grid search to find the model parameters of a neural network model, establishing a neural network model, fitting the neural network model according to the training set, adjusting the hyperparameters of the neural network model according to the root mean square error of the test set, substituting the test set into the adjusted neural network model and calculating the accuracy rate, and when the accuracy rate reaches a preset accuracy rate threshold, determining the specular mask image according to the obstacle avoidance target image.

[0064] Specifically, the acquired image dataset includes key data such as sunlight area images during various sunlight reflections in the current environment, a three-dimensional coordinate system, and point cloud environment data. These data record the sunlight reflection conditions at different time periods. The image dataset is divided into a training set and a test set. Usually, 70% - 80% of the data is used as the training set, and the rest is used as the test set. Ensure that both the training set and the test set contain data on various sunlight reflection conditions to improve the generalization ability of the model. Cross-validation and Grid Search are used to find the model parameters of the neural network model. Cross-validation divides the data into several parts and trains the model multiple times to verify its stability and performance. Grid Search exhaustively searches parameter combinations in the parameter space to establish the neural network model. The neural network model contains multiple layers, different types of neurons, and activation functions, aiming to capture complex relationships in the data. When the test set is substituted into the neural network model, the root mean square error of the test set is obtained, and based on this, the hyperparameters of the neural network model are adjusted to optimize the generalization ability of the model, enabling it to maintain good prediction performance on unseen data. Then, the test set is substituted into the adjusted neural network model to calculate the accuracy. Accuracy is an important indicator for evaluating the model's performance. After the model reaches the preset accuracy threshold, it indicates that the model's performance has remained stable, and it is considered that the model has reached a satisfactory performance level and can determine the reflective mask image based on the image of the obstacle to be avoided.

[0065] In some embodiments of the present application, when extracting the pixel values of the reflective mask image and determining whether there is a reflective area based on the pixel values and the obstacle point cloud data, it includes: extracting all the point cloud data points corresponding in the obstacle point cloud data, extracting all the mask pixel points corresponding to the reflective mask image, and determining the pixel values of all the mask pixel points. When the pixel value of the mask pixel point corresponding to the point cloud data point is 1, the point cloud data point is generated as a reflective mark. When the pixel value of the mask pixel point corresponding to the point cloud data point is 0, the point cloud data point is generated as a non-reflective mark. If a reflective mark is recognized, it is determined that there is a reflective area. If no reflective mark is recognized, it is determined that there is no reflective area.

[0066] Specifically, since the reflective mask image is determined based on the image of the obstacle to be avoided and the registration operation has been performed when determining the image of the obstacle to be avoided, therefore, the reflective mask image and the image of the obstacle to be avoided maintain the coordinate system of the same point cloud environment, and the point cloud data points corresponding in the obstacle point cloud data and the mask pixel points corresponding to the reflective mask image are in one-to-one correspondence. The reflective mask image is a binary image, and the pixel values of the mask pixel points in it are only 0 or 1. 0 represents non-reflective pixel points, and 1 represents reflective pixel points. By finding the pixel value of the mask pixel point at the corresponding position of the point cloud data point, it is possible to dynamically determine whether there is interference from sunlight reflection in the point cloud data point, ensuring the stability and accuracy of obstacle avoidance.

[0067] In some embodiments of the present application, when determining the first obstacle map based on the obstacle point cloud data, it includes: projecting all the point cloud data points generating non-light marks onto the occupancy grid map, and based on the position information of each point cloud data point, determining whether there are point cloud data points with non-light marks in each grid cell of the occupancy grid map to determine the first obstacle map. When there are point cloud data points with non-light marks in a grid cell, mark this grid cell as occupied by an obstacle; when there are no point cloud data points with non-light marks in a grid cell, mark this grid cell as not occupied by an obstacle, and determine the first obstacle map based on the occupied and non-occupied states of the obstacles.

[0068] Specifically, the occupancy grid map divides the space of the current environment into several grid cells, preferably 30 grid cells. After the point cloud data points generating non-light marks are projected onto the occupancy grid map, the spatial position of each point cloud data point will correspond to one of the grid cells. By determining whether there are point cloud data points with non-light marks in each grid cell, the state of this grid cell can be determined. When there are point cloud data points with non-light marks in a grid cell, it indicates that there is an obstacle at this position, and this grid cell is marked as occupied by an obstacle; otherwise, it is marked as not occupied by an obstacle. Based on the states of all grid cells, the first obstacle map reflecting the distribution of obstacles in the environment can be constructed, which is applicable to various complex environments. Regardless of the shape, size, and distribution of the obstacles, it can effectively identify and mark them, improving the stability and flexibility of obstacle avoidance.

[0069] In some embodiments of the present application, when deleting the obstacle point cloud data according to the reflective area to obtain the light-avoiding point cloud data and determining the radar point cloud data according to the ultrasonic radar, it includes: deleting the point cloud data points generating reflective marks, and determining the remaining point cloud data points generating non-light marks as the light-avoiding point cloud data, obtaining the round-trip time of the echo of the ultrasonic radar, determining the distance information according to the round-trip time, converting the distance information into three-dimensional space coordinates, constructing a radar coordinate system, converting each coordinate point of the radar coordinate system in the point cloud data format to determine the radar data points, and determining all the radar data points as the radar point cloud data.

[0070] Specifically, in the current environment, the reflective area interferes with the accuracy of the obstacle point cloud data. By deleting the point cloud data points that generate the reflective markers, the interference of reflection is eliminated, and the remaining point cloud data points without light markers are determined as the light-avoiding point cloud data, improving the accuracy of the light-avoiding point cloud data in reflecting the current environmental conditions, avoiding the noise generated by the reflective area from interfering with subsequent judgments, and thus improving the accuracy of obstacle avoidance. The ultrasonic radar detects by emitting ultrasonic waves and receiving the echoes reflected from obstacles. There is a fixed mathematical relationship between the round-trip time of the echo and the distance, and based on this, the distance information of the obstacle can be calculated. The distance information is converted into three-dimensional space coordinates. At this time, the coordinate system corresponding to the three-dimensional space coordinates is translated or transformed by a matrix so that the constructed radar coordinate system is consistent with the coordinate system of the point cloud environment. Then, each coordinate point is transformed according to the point cloud data format to form radar data points, and finally, radar point cloud data is obtained. The techniques of translation or matrix transformation, and the conversion of coordinate points into the point cloud data format are cumbersome and mature, and will not be introduced in detail here. By removing the point cloud data points in the reflective area, the interference of factors such as sunlight reflected by the ground on the data is effectively avoided. Moreover, the ultrasonic radar makes up for the removed point cloud data points of the reflective markers. By combining the light-avoiding point cloud data and the radar point cloud data, the advantages of different sensors are fully utilized, the comprehensiveness of environmental information is improved, the deficiencies of a single sensor are made up for, the reliability and stability of obstacle avoidance are enhanced, and the risk of obstacle avoidance failure caused by data errors is reduced.

[0071] In some embodiments of the present application, when fusing the light-avoiding point cloud data and the radar point cloud data to determine the target obstacle point cloud data, it includes: performing a coincidence comparison between the light-avoiding point cloud data and the radar point cloud data to determine the coincident points and non-coincident points with overlapping positions. Based on the DBSCAN algorithm, with each coincident point as the center, a neighborhood radius is set, and all the clustering data points within this neighborhood are searched and extracted. The clustering data points include radar data points and the point cloud data points that generate the lightless markers. A minimum number of points is preset, the number of data points of the clustering data points within the neighborhood radius of each coincident point is obtained, and the number of data points is compared with the minimum number of points to determine whether this coincident point is a core point.

[0072] Specifically, since the constructed radar coordinate system is consistent with the coordinate system of the point cloud environment, the point cloud data points included in the light-avoiding point cloud data are consistent with the positions of the radar data points included in the radar point cloud data. By overlapping and comparing the light-avoiding point cloud data and the radar point cloud data, the overlapping points and non-overlapping points with overlapping positions can be distinguished. The overlapping points indicate that different sensors detect the same obstacle at this position. Since the ultrasonic radar is not affected by sunlight reflection, the non-overlapping points reflect the point cloud data points for deleting and generating reflective markers, and the corresponding supplement made by the ultrasonic radar for this deleted part. During the fusion process, in-depth analysis of the overlapping points is carried out to avoid redundancy or errors caused by simple data superposition, so that the fused data can accurately reflect the actual distribution of obstacles. Based on the density characteristics of the DBSCAN algorithm, the noise points in the low-density area can be effectively identified and filtered out, improving the quality of the target obstacle point cloud data, enhancing the adaptability and reliability of obstacle avoidance, and providing reliable data support for subsequent obstacle avoidance path planning.

[0073] In some embodiments of the present application, when comparing the number of data points with the minimum number of points to determine whether the overlapping point is a core point, it includes: when the number of data points is greater than or equal to the minimum number of points, determining the overlapping point as a core point, expanding its neighborhood radius, and determining the clustering set; when the number of data points is less than the minimum number of points and the clustering data points belong to the neighborhood radius determined as the core point, determining the overlapping point as a boundary point; when the number of data points is less than the minimum number of points and the clustering data points do not belong to the neighborhood radius determined as the core point, determining the overlapping point as a noise point.

[0074] In some embodiments of the present application, the obstacle avoidance method for multi-sensor fusion further includes: merging the clustering set, boundary points, and non-overlapping points to determine the target obstacle point cloud data.

[0075] Specifically, based on the DBSCAN algorithm, when the number of data points within the neighborhood of a coincident point is greater than or equal to the minimum number of points, indicating a densely distributed point cloud within that neighborhood, the point cloud is identified as a core point. By expanding the neighborhood radius of the core point, more adjacent clustered data points can be incorporated, thereby defining a cluster set and, in turn, determining features such as the main outline of the obstacle. When the number of data points is less than the minimum number of points, but the clustered data points are within the neighborhood radius of the identified core point, these clustered data points, despite their own low neighborhood density, are related to the neighborhood of the core point and are therefore identified as boundary points, representing the edge of the obstacle. Clustered data points with both less than the minimum number of points and not within the neighborhood radius of the core point are considered noise points due to their low density and isolated location. The cluster set, boundary points, and non-coinciding points are then merged. The cluster set reflects the main structure of the obstacle, the boundary points provide additional information about the obstacle's edges, and the non-coinciding points include supplementary data from the ultrasonic sensor. This merging of the three comprehensively integrates multi-sensor data to determine the target obstacle point cloud. Through reasonable classification and merging, the integrity of the target obstacle point cloud data is ensured, thereby accurately depicting the overall picture of the obstacle and avoiding missing information. In addition, by identifying noise points and not merging them, the impact of environmental interference and sensor errors is effectively reduced, and the reliability of obstacle avoidance is improved.

[0076] It can be understood that the method of determining the second obstacle map based on the target obstacle point cloud data is the same as the method of determining the first obstacle map based on the obstacle point cloud data, except that the point cloud data points are different. Therefore, the description of the method of determining the second obstacle map is redundant and will not be repeated here.

[0077] In summary, the beneficial effects of the present invention are as follows: the image of the target obstacle to be avoided is determined based on a number of images of the obstacle to be avoided, and obstacle point cloud data is generated accordingly, thereby ensuring the comprehensiveness of the environmental information obtained. With the help of the sunlight segmentation model, the reflective mask image can be determined based on the image of the target obstacle to be avoided. The interference of the 3D camera in the sunlight reflection environment is effectively identified, and the noise generated by the reflection that affects the stability of the obstacle avoidance is avoided, thereby improving the reliability and adaptability of the obstacle avoidance. The first obstacle map is generated based on the obstacle point cloud data, ensuring the timeliness of the map drawing. In the complex situation of sunlight reflection, by deleting the obstacle point cloud data affected by the reflection, the light avoidance point cloud data is determined, and the radar point cloud data of the ultrasonic radar is integrated to generate the second obstacle map. This not only overcomes the limitations of the 3D camera in the reflective environment, but also combines the stable ranging characteristics of the ultrasonic radar to achieve the complementary advantages of the two sensors, thereby providing a reliable basis for obstacle avoidance decision-making. The multi-sensor fusion method no longer relies on a single sensor, enhancing the stability and adaptability of the overall obstacle avoidance process.

[0078] In another preferred embodiment based on the above embodiment, refer toFigure 2 As shown in Figure 2 , this embodiment provides an obstacle avoidance system with multi-sensor fusion, which is applied to the above-mentioned obstacle avoidance method with multi-sensor fusion, and includes:

[0079] An acquisition module, configured to obtain a plurality of images to be avoided, process the plurality of images to be avoided, determine a target image to be avoided, and determine obstacle point cloud data according to the target image to be avoided;

[0080] A judgment module, configured to determine a reflective mask image based on a sunlight segmentation model according to the target image to be avoided, extract the pixel values of the reflective mask image, and judge whether there is a reflective area according to the pixel values and the obstacle point cloud data;

[0081] A processing module, configured to determine a first obstacle map according to the obstacle point cloud data if there is no reflective area, and if there is a reflective area, delete the obstacle point cloud data according to the reflective area to obtain avoidance point cloud data, determine radar point cloud data according to the ultrasonic radar, fuse the avoidance point cloud data and the radar point cloud data to determine target obstacle point cloud data, and determine a second obstacle map according to the target obstacle point cloud data;

[0082] An obstacle avoidance module, configured to determine an obstacle avoidance path based on the first obstacle map or the second obstacle map.

[0083] Specifically, determining the target image to be avoided according to a plurality of images to be avoided and generating obstacle point cloud data accordingly ensures the comprehensiveness of obtaining environmental information. With the help of the sunlight segmentation model, a reflective mask image can be determined according to the target image to be avoided. It effectively identifies the interference of the 3D camera in the sunlight reflection environment, avoids the noise caused by reflection, affects the stability of obstacle avoidance, and improves the reliability and adaptability of obstacle avoidance. Generating the first obstacle map according to the obstacle point cloud data ensures the timeliness of map drawing. In the complex situation of sunlight reflection, by deleting the obstacle point cloud data affected by reflection to determine the avoidance point cloud data and fusing the radar point cloud data of the ultrasonic radar to generate the second obstacle map, it not only overcomes the limitations of the 3D camera in the reflective environment, but also combines the stable ranging characteristics of the ultrasonic radar, realizes the complementary advantages of the two sensors, and thus provides a reliable basis for obstacle avoidance decision-making. The multi-sensor fusion method no longer relies on a single sensor, enhancing the stability and adaptability of the overall obstacle avoidance process.

[0084] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0085] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of flows and / or blocks in the flowchart and / or block diagram can also be implemented. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one or more flows and / or Figure 1 blocks.

[0086] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in Figure 1 one or more flows and / or Figure 1 blocks.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, so that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process. Therefore, the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in Figure 1 one or more flows and / or Figure 1 blocks.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: the specific implementation manners of the present invention can still be modified or equivalently replaced, and any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the protection scope of the claims of the present invention.

Claims

1. A multi-sensor fusion obstacle avoidance method, characterized in that, Including: Obtain a plurality of images to be obstacle-avoided, process the plurality of images to be obstacle-avoided, determine an obstacle-avoidance target image, and determine obstacle point cloud data according to the obstacle-avoidance target image; According to the obstacle-avoidance target image, determine a specular reflection mask image based on a sunlight segmentation model, extract pixel values of the specular reflection mask image, and determine whether there is a specular reflection area according to the pixel values and the obstacle point cloud data; When there is no such specular reflection area, determine a first obstacle map according to the obstacle point cloud data. When there is such a specular reflection area, delete the obstacle point cloud data according to the specular reflection area to obtain specular reflection avoidance point cloud data, and determine radar point cloud data according to an ultrasonic radar. Integrate the specular reflection avoidance point cloud data and the radar point cloud data to determine target obstacle point cloud data, and determine a second obstacle map according to the target obstacle point cloud data; Determine an obstacle-avoidance path based on the first obstacle map or the second obstacle map; Extract all point cloud data points corresponding in the obstacle point cloud data, extract all mask pixel points corresponding to the specular reflection mask image, and determine pixel values of all mask pixel points; When the pixel value of the mask pixel point corresponding to the point cloud data point is 1, generate a specular reflection mark for the point cloud data point; When the pixel value of the mask pixel point corresponding to the point cloud data point is 0, generate a non-specular reflection mark for the point cloud data point; Project all the point cloud data points generated with non-specular reflection marks onto an occupancy grid map, and determine whether there are point cloud data points with non-specular reflection marks in each grid cell in the occupancy grid map according to the position information of each point cloud data point, so as to determine the first obstacle map; When there are point cloud data points with non-specular reflection marks in a grid cell, mark the grid cell as an occupied state by an obstacle; When there are no point cloud data points with non-specular reflection marks in a grid cell, mark the grid cell as a non-occupied state by an obstacle; Determine the first obstacle map according to the occupied state by an obstacle and the non-occupied state by an obstacle; Delete the point cloud data points that generate the specular reflection marks, and determine the remaining point cloud data points that generate the non-specular reflection marks as the specular reflection avoidance point cloud data; Obtain the round-trip time of the echo of the ultrasonic radar, determine distance information according to the round-trip time of the echo, convert the distance information into three-dimensional space coordinates, and construct a radar coordinate system.

2. The obstacle avoidance method for multi-sensor fusion according to claim 1, wherein When processing a plurality of images to be obstacle-avoided to determine an obstacle-avoidance target image, it includes: Perform preprocessing on the plurality of images to be obstacle-avoided, and the preprocessing includes denoising and geometric correction; Extract feature points from the preprocessed plurality of images to be obstacle-avoided according to the accelerated robust feature algorithm, and use the RANSAC algorithm to match the extracted feature points to determine the image correlation data between the preprocessed plurality of images to be obstacle-avoided; Based on the image correlation data, register the preprocessed plurality of images to be obstacle-avoided, and merge the registered plurality of images to be obstacle-avoided according to multi-band fusion to determine the obstacle-avoidance target image.

3. The multi-sensor fusion obstacle avoidance method according to claim 2, wherein When determining a specular reflection mask image based on a sunlight segmentation model according to the obstacle-avoidance target image, it includes: Obtain an image dataset, divide the image dataset into a training set and a test set, use cross-validation and grid search to find the model parameters of a neural network model, and establish a neural network model; Fit the neural network model according to the training set, and adjust the hyperparameters of the neural network model according to the root mean square error of the test set; Substitute the test set into the adjusted neural network model and calculate the accuracy. When the accuracy reaches the preset accuracy threshold, determine the reflective mask image according to the target image to be avoided; 4. The multi-sensor fusion obstacle avoidance method according to claim 3, characterized in that, When extracting the pixel values of the reflective mask image and judging whether there is a reflective area according to the pixel values and the obstacle point cloud data, it includes: If the reflective mark is recognized, it is determined that there is a reflective area; If the reflective mark is not recognized, it is determined that there is no reflective area; 5. The obstacle avoidance method for multi-sensor fusion according to claim 4, characterized in that, When deleting the obstacle point cloud data according to the reflective area to obtain the light-avoiding point cloud data, and determining the radar point cloud data according to the ultrasonic radar, it includes: Convert each coordinate point in the radar coordinate system into the point cloud data format to determine the radar data points, and determine all the radar data points as the radar point cloud data; 6. The multi-sensor fusion obstacle avoidance method according to claim 5, wherein When fusing the light-avoiding point cloud data and the radar point cloud data to determine the target obstacle point cloud data, it includes: Perform a coincidence comparison between the light-avoiding point cloud data and the radar point cloud data to determine the coincident points and non-coincident points with coincident positions; Based on the DBSCAN algorithm, with each coincident point as the center, set the neighborhood radius, and search and extract all the clustering data points within the neighborhood. The clustering data points include the radar data points and the point cloud data points generating the non-light mark; Preset the minimum number of points, obtain the number of data points of the clustering data points within the neighborhood radius of each coincident point, and compare the number of data points with the minimum number of points to determine whether the coincident point is a core point; 7. The obstacle avoidance method based on multi-sensor fusion according to claim 6, characterized in that, When comparing the number of data points with the minimum number of points to determine whether the coincident point is a core point, it includes: When the number of data points is greater than or equal to the minimum number of points, determine the coincident point as a core point, expand its neighborhood radius, and determine the clustering set; When the number of data points is less than the minimum number of points and the clustering data points belong to the neighborhood radius determined as a core point, determine the coincident point as a boundary point; When the number of data points is less than the minimum number of points and the clustering data points do not belong to the neighborhood radius determined as a core point, determine the coincident point as a noise point; 8. The multi-sensor fusion obstacle avoidance method according to claim 7, wherein It also includes: Merge the clustering set, the boundary points and the non-coincident points to determine the target obstacle point cloud data; 9. An obstacle avoidance system based on multi-sensor fusion, which is applied to the multi-sensor fusion obstacle avoidance method according to any one of claims 1-8, characterized in that, It includes: An acquisition module, configured to acquire a plurality of images to be avoided, process the plurality of images to be avoided, determine a target image to be avoided, and determine obstacle point cloud data according to the target image to be avoided; A judgment module, configured to determine a reflective mask image based on a sunlight segmentation model according to the target image to be avoided, extract the pixel values of the reflective mask image, and judge whether there is a reflective area according to the pixel values and the obstacle point cloud data; The processing module is configured to determine a first obstacle map according to the obstacle point cloud data when the reflective area does not exist, and when the reflective area exists, delete the obstacle point cloud data according to the reflective area to obtain light-avoiding point cloud data, determine radar point cloud data according to the ultrasonic radar, fuse the light-avoiding point cloud data and the radar point cloud data to determine target obstacle point cloud data, and determine a second obstacle map according to the target obstacle point cloud data; The obstacle avoidance module is configured to determine an obstacle avoidance path based on the first obstacle map or the second obstacle map.

Images