Obstacle classification method and related apparatus

Abstract

The application relates to an obstacle classification method and a related device, comprising: obtaining a grid result corresponding to a surrounding area of a current position of a target device based on data collected by at least one sensor on the target device, each grid in the grid result having category information and a height value; determining candidate category grids from the grid result according to the category information and the height value; clustering the candidate category grids to obtain a plurality of candidate category grid clusters; classifying the candidate category grid clusters based on a preset classification condition and a preset reference semantic map to determine a first category obstacle and a second category obstacle, and the preset classification condition is associated with position information and height values of the candidate category grid clusters. The scheme provided by the application can process data obtained by a sensor, classify obstacles in a surrounding area, and improve the accuracy of obstacle classification and the safety of vehicle driving.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to an obstacle classification method and a related device thereof. BACKGROUND

[0002] The open-pit mine road environment is complex, often accompanied by mine obstacles such as stone piles, soil slopes, deep pits, retaining walls, pedestrians, and auxiliary operation vehicles. In order to prevent mine cars driving on the mine road from colliding with mine obstacles, mine obstacle detection needs to be performed on the mine road where the mine cars are driving.

[0003] In related technologies, data obtained by a single sensor is generally used to classify obstacles. However, in a mine environment, unknown types of obstacles such as ore, gravel, and small parts may exist. The existing detection method is difficult to detect different types of obstacles, resulting in low obstacle detection accuracy and difficulty in ensuring the driving safety of vehicles. SUMMARY

[0004] To solve or partially solve the problems in related technologies, the present application provides an obstacle classification method and a related device thereof, which can process data obtained by a sensor, classify obstacles in a surrounding area, and improve the accuracy of obstacle classification and the safety of vehicle driving.

[0005] The first aspect of the present application provides an obstacle classification method, including obtaining a grid result corresponding to a surrounding area of a current position of a target device based on data collected in real time by at least one sensor on the target device, each grid in the grid result having category information and a height value; determining a candidate category grid from the grid result according to the category information and the height value; clustering the candidate category grid to obtain a plurality of candidate category grid clusters, wherein the candidate category grid clusters include at least one candidate category grid; classifying the candidate category grid clusters based on a preset classification condition and a preset reference semantic map to determine a first category obstacle and a second category obstacle, the preset classification condition being associated with position information and a height value of the candidate category grid cluster, and the preset reference semantic map including the surrounding area of the current position of the target device.

[0006] In combination with the first aspect, in a possible implementation manner of the first aspect, the obtaining of the grid result corresponding to the surrounding area of the current position of the target device based on the data collected in real time by the at least one sensor on the target device includes: inputting the data collected in real time by the at least one sensor into a multi-modal neural network model to obtain the grid result output by the multi-modal neural network model; and the data collected in real time by the sensor includes point cloud data, image data, and radar data of the surrounding area of the current position of the target device.

[0007] With reference to the first aspect, in a possible implementation form of the first aspect, the method further includes: determining a first category grid from the grid result according to the category information of each grid; and displaying a third category obstacle in the reference semantic map based on the category information of the first category grid; wherein the third category obstacle includes a vehicle and a sandstorm.

[0008] With reference to the first aspect, in a possible implementation form of the first aspect, the determining the candidate category grid from the grid result according to the category information and the height value includes: determining a second category grid from the grid result according to the category information of each grid; performing a sliding average processing on an initial height value of the second category grid to obtain an average height value of the second category grid; determining the second category grid with the average height value greater than a first preset height threshold as the candidate category grid; and determining the second category grid with the average height value not greater than the first preset height threshold as a ground category grid.

[0009] With reference to the first aspect, in a possible implementation form of the first aspect, the performing the sliding average processing on the height value of the second category grid to obtain the average height value of the second category grid includes:

[0010] The formula for the sliding average processing on the second category grid includes:

[0011]

[0012] wherein, the target weight is positively correlated with an observation number of a current grid, the average height value of the current grid, the initial height value of the current grid. With reference to the first aspect, in a possible implementation form of the first aspect, the clustering the candidate category grid to obtain a plurality of candidate category grid clusters includes: performing island filtering on the candidate category grid to obtain a plurality of candidate category grid clusters; generating a minimum bounding rectangle of each candidate category grid cluster and obtaining a diagonal length of the minimum bounding rectangle; determining a first target grid cluster from the candidate category grid clusters with the diagonal length not greater than a preset length threshold; and determining a second target grid cluster from the candidate category grid clusters with the diagonal length greater than the preset length threshold.

[0013]

[0014] ​In a possible implementation of the first aspect, the classifying the candidate category grid cluster based on the preset classification condition and the preset reference semantic map includes: if the first target grid cluster is located on one side of a lane in the reference semantic map, and the ground category grid exists within a preset distance range of the first target grid cluster, and an average height value of the first target grid cluster is greater than a second preset height value, the first target grid cluster is determined as the second category obstacle, and the second category obstacle is displayed in the reference semantic map; if the first target grid cluster is located on the lane in the reference semantic map, and the ground category grid does not exist within the preset distance range of the first target grid cluster, and the average height value of the first target grid cluster is greater than a third preset height value, the first target grid cluster is determined as the first category obstacle, and the first category obstacle is displayed in the reference semantic map; and the second target grid cluster is determined as the second category obstacle, and the second category obstacle corresponding to the second target grid cluster is displayed in the reference semantic map.

[0015] In a possible implementation of the first aspect, the first category obstacle is gravel, and the second category obstacle is a soil wall.

[0016] The second aspect of the present application provides an obstacle classification device, including: an acquisition module configured to obtain a grid result corresponding to a surrounding area of a current position of a target device based on data collected by at least one sensor on the target device in real time, each grid in the grid result being provided with category information and a height value; a determination module configured to determine candidate category grids from the grid result according to the category information and the height value; a clustering module configured to cluster the candidate category grids to obtain a plurality of candidate category grid clusters, wherein each candidate category grid cluster includes at least one candidate category grid; and a classification module configured to classify the candidate category grid clusters based on a preset classification condition and a preset reference semantic map to obtain a first category obstacle and a second category obstacle, the preset classification condition being associated with position information and a height value of the candidate category grid cluster, and the preset reference semantic map including the surrounding area of the current position of the target device.

[0017] The third aspect of the present application provides an electronic device, including:

[0018] a processor; and

[0019] a memory having executable code stored thereon, the executable code, when executed by the processor, causing the processor to perform the method described above.

[0020] The fourth aspect of the present application provides a computer readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0021] The technical solution provided by the present application can include the following beneficial effects:

[0022] The obstacle classification method and the related device provided by the present application include the following steps: obtaining a grid result corresponding to a surrounding area of a current position of a target device based on data collected by at least one sensor on the target device, each grid in the grid result having category information and a height value; determining candidate category grids from the grid result according to the category information and the height value; clustering the candidate category grids to obtain a plurality of candidate category grid clusters, wherein each candidate category grid cluster includes at least one candidate category grid; and classifying the candidate category grid clusters based on a preset classification condition and a preset reference semantic map to determine a first category obstacle and a second category obstacle, wherein the preset classification condition is associated with position information and height values of the candidate category grid clusters, the preset reference semantic map includes the surrounding area of the current position of the target device, can process data collected by the sensor, and can classify obstacles in the surrounding area, thereby improving the accuracy of obstacle classification and the safety of vehicle driving.

[0023] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present application. BRIEF DESCRIPTION OF DRAWINGS

[0024] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings, in which like reference characters refer to like parts throughout the figures, and wherein:

[0025] Figure 1 is a flowchart of an obstacle classification method according to an embodiment of the present application;

[0026] Figure 2 is a structural diagram of an obstacle classification device according to an embodiment of the present application;

[0027] Figure 3 is a structural diagram of an electronic device according to an embodiment of the present application;

[0028] Figure 4 is a reference semantic map including a first category obstacle and a second category obstacle according to an embodiment of the present application;

[0029] Figure 5 is a diagram of single-frame data collected by a target device according to an embodiment of the present application;

[0030] Figure 6 is a superimposed schematic diagram of the reference semantic map and the single-frame data collected by the target device shown in the embodiments of the present application. DETAILED DESCRIPTION

[0031] Embodiments of the present application will be described in more detail with reference to the drawings. Although the embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application is more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0032] The terminology used in the present application is merely for the purpose of describing particular embodiments and is not intended to limit the present application. As used in the present application and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the term "and / or" used herein as a transitional phrase is intended to mean and include any and all possible combinations of one or more associated listed items.

[0033] It should be understood that although the terms "first", "second", "third", etc. can be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information without departing from the scope of the present application. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0034] In a complex operation environment such as a mine, there are often a large number of obstacles, and the mine environment has the characteristics of complex terrain, various types of obstacles and dynamic changes, such as different vehicles, sand, etc. On the driving path of the vehicle, there are often obstacles such as gravel, sand, and earth walls. The traditional obstacle detection method can detect the earth wall on the driving path and display it on the map to let the vehicle avoid the earth wall. However, due to the low height and small size of small objects such as sand and gravel, these small objects are easily detected as the ground during detection, and this type of obstacle is not displayed on the map. Especially when sand and gravel exist on the driving path of the vehicle, the vehicle cannot detect this type of obstacle, which makes the vehicle unable to avoid small objects such as sand and gravel during driving, causing danger to the driving of the vehicle.

[0035] To solve the above problems, the embodiment of the present application provides an obstacle classification method and a related device, which can process the data collected by the sensor, classify the obstacles in the surrounding area, and improve the accuracy of obstacle classification and the safety of vehicle driving.

[0036] The technical solutions of the embodiments of the present application are described in detail below with reference to the drawings.

[0037] Figure 1 is a flowchart of the obstacle classification method shown in the embodiments of the present application.

[0038] Referring to Figure 1 and Figures 4-6 , an obstacle classification method comprises:

[0039] S110: obtaining a grid result corresponding to the surrounding area of the current position of the target device based on the data collected by at least one sensor on the target device, each grid in the grid result having category information and a height value.

[0040] Specifically, the target device can be a vehicle, and the current working environment of the target device can be a mine environment. The target device can include multiple sensors, which can be multi-source sensors. The data collected by the multiple sensors in real time can be multi-source data. For example, the at least one sensor can include a laser radar, which can be used to collect laser radar data in real time. The at least one sensor can also include a vision sensor, which can be used to collect vision data in real time. As another example, the vision sensor can be a camera. The at least one sensor can also include other specific implementations, which are not limited.

[0041] Specifically, the grid result refers to the process of converting three-dimensional point cloud data into a two-dimensional grid matrix. Each grid records the category information and the corresponding height value of the obstacles in the area. The grid result can be a 0.25 resolution grid map. For example, when the category information of the grid is a vehicle, it can be determined that there is a vehicle in the real area corresponding to the grid.

[0042] For example, the surrounding area of the current position of the target device can be an area within a preset range in front of and behind the current position of the target device. For example, the preset range can be 100 meters or 200 meters, or other values. For example, the surrounding area is an area within 100 meters in front of and 100 meters behind the current position of the target device. Therefore, the specific implementation of the surrounding area is not limited.

[0043] In a possible implementation, the grid result corresponding to the surrounding area of the current position of the target device is obtained based on data collected in real time by at least one sensor on the target device, including: inputting the data collected in real time by the at least one sensor into a multi-modal neural network model to obtain a grid result output by the multi-modal neural network model; wherein the data collected in real time by the sensor includes point cloud data, image data, and radar data of the surrounding area of the current position of the target device, the target device can be a movable device such as a mining truck, a shovel, a bulldozer, a small vehicle, or other types of vehicles, and can also be a drone or a robot, etc., and no specific limitation is made to this.

[0044] Specifically, the multi-modal neural network model can be a multi-task multi-sensor fusion model with unified bird's-eye view representation (BEVFusion), which can fuse multi-modal sensor data (such as cameras and lidar) through a unified bird's-eye view (BEV) space, and support multi-task perception (such as 3D target detection, BEV map segmentation, etc.). The target neural network model can also be other types of neural network models, for example, a neural network model including a convolutional neural network and a point cloud processing network, and no specific limitation is made to this.

[0045] Specifically, the sensors on the target device include multiple sensors, and point cloud data, image data, and radar data can be obtained through the multiple sensors. The data collected in real time by the multiple sensors can be subjected to feature extraction and feature fusion in the multi-modal neural network model, and after the collected data is subjected to feature extraction and feature fusion, a perception result of the surrounding area of the current position of the target device output by the multi-modal neural network model is obtained, that is, the above-mentioned grid result.

[0046] Exemplarily, the point cloud data, the image data and the radar data collected by the sensor in real time can be synchronously input into a multi-modal neural network model for processing. The multi-modal neural network model first extracts features of each modality through a dedicated branch, for example, uses a PointNet++ network to process the point cloud data, a ResNet to process the image data, and a one-dimensional convolution network to process the radar data, and then aligns and splices the multi-modal features through a feature fusion module to generate a comprehensive feature representation containing spatial, texture and dynamic information. The comprehensive feature passes through a decoder module to generate a grid result, each grid containing category information and height value. The point cloud data can ensure spatial positioning accuracy, the image data can enhance category recognition accuracy, and the radar data can provide basic perception ability in a mine environment. Through multi-modal fusion of the point cloud data, the image data and the radar data, environmental interference factors can be effectively excluded, which is particularly suitable for mine scenes with a large amount of dust, vibration and complex terrain, can avoid misjudgment of the category of obstacles due to failure of a single sensor, and the complementary features of three-dimensional point cloud and two-dimensional image can significantly reduce the probability of misjudging ground concave-convex as obstacles.

[0047] S120: determining a candidate category grid from the grid result according to the category information and the height value.

[0048] Specifically, the category information of the grid can include vehicles, earth walls, sandstorms, etc., and the grid also includes a height value. The candidate category grid can be a potential obstacle area filtered by a height threshold. For example, when the category information of the grid is a vehicle and the height value is greater than the height threshold, the grid can be determined as the candidate category grid. Here, no limitation is made. By filtering the candidate category grid, the subsequent classification of the obstacle corresponding to the grid can be better performed.

[0049] In a possible implementation, the method further includes: determining a first category grid from the grid result according to the category information of each grid; and displaying a third category obstacle in the reference semantic map based on the category information of the first category grid; wherein the third category obstacle includes vehicles and sandstorms.

[0050] Specifically, the first category grid refers to the grid in the grid result corresponding to the vehicle or the sandstorm, and the grid cluster of the first category grid aggregated by clustering the first category grid is obtained, and the grid cluster is determined as the third category obstacle. The boundary coordinates of the grid cluster are mapped to the corresponding position of the reference semantic map. For example, the grid cluster corresponding to the vehicle is labeled as a vehicle on the reference semantic map, and the grid cluster corresponding to the sandstorm is labeled as a dynamic environmental interference. The reference semantic map displays the position and distribution range of the vehicle and the sandstorm on the reference semantic map by superimposing real-time obstacle information. By directly extracting the category information of the corresponding grid of the vehicle and the sandstorm and associating the reference semantic map, the vehicle and the sandstorm can be effectively distinguished from dynamic obstacles such as sandstorm in the mine scene, and the working safety of the target device in the mine environment is improved.

[0051] In a possible implementation, the second category grid is determined from the grid result according to the category information of each grid; the initial height value of the second category grid is subjected to a sliding average processing to obtain an average height value of the second category grid; the second category grid with the average height value greater than a first preset height threshold is determined as a candidate category grid; and the second category grid with the average height value not greater than the first preset height threshold is determined as a ground category grid.

[0052] Specifically, the second category grid refers to the grid preliminarily classified as a soil wall category and a ground category by the multi-modal model. The average height value of the second category grid can be obtained by subjecting the initial height value of the second category grid to a sliding average processing. The sliding average processing refers to a weighted average calculation method based on a time sequence, which can be used to eliminate single-frame data noise interference. The obtained average height value can be better used for subsequent classification of the second category grid.

[0053] Specifically, the first preset height threshold can be a critical height value for the ground and the obstacle. The first preset height threshold can be dynamically set according to the terrain undulation characteristics of the mine environment. For example, the first preset height threshold can be set to a range of 0.3 meters to 0.5 meters. The second category grid higher than the first preset height threshold can be determined as the candidate category grid, which can be the soil wall. The second category grid not greater than the first preset height threshold can be determined as the ground category grid. The ground category grid and the reference semantic map can be superimposed in real time to display the position and distribution range of the ground on the reference semantic map. The obstacles can be dynamically distinguished according to the height of the obstacle, and the robustness of the obstacle detection in the mine scene is improved.

[0054] In a possible implementation, the height value of the second category grid is subjected to a sliding average processing to obtain an average height value of the second category grid, including:

[0055] The formula for the sliding average processing of the second category grid includes:

[0056]

[0057] wherein, is a target weight, the target weight is positively correlated with the number of observations of the current grid, is an average height value of the current grid, is an initial height value of the current grid.

[0058] Specifically, the target weight refers to a coefficient associated with the number of observations, which can specifically map the number of observations to a weight value in the range of 0 to 1 through a linear or nonlinear function, for example, the weight value increases by 0.05 for each increase in the number of observations, and the average height value is obtained by accumulating multiple frames of data.

[0059] Specifically, in a mine environment, the initial height value collected by the sensor may fluctuate abnormally due to noise or transient occlusion. By introducing a moving average formula, the initial height value of the current frame is superimposed with the historical average height value according to the weight, and the weight coefficient is dynamically adjusted according to the number of observations of the grid. For example, when a certain grid is detected for the first time, the number of observations is 1, and the weight value can be set to 0.2. As the number of observations increases to 5, the weight value gradually increases to 0.5. The average height value of the grid obtained by the moving average calculation can be more accurate, can effectively eliminate the transient noise interference in a single frame of data, solve the problem of unstable height measurement caused by dust, equipment vibration and other factors in the mine scene, and avoid false detection of obstacles caused by height misjudgment.

[0060] S130: clustering the candidate category grids to obtain a plurality of candidate category grid clusters, wherein each candidate category grid cluster includes at least one candidate category grid.

[0061] Specifically, after obtaining a plurality of candidate category grids, the candidate category grids can be clustered to obtain a candidate category grid cluster. The candidate category grid cluster can be an obstacle aggregation area formed by connected component analysis. When the candidate category grid cluster is formed, the candidate category grid cluster can be pre-provided with category information. For example, when the category information of the candidate category grid is a soil wall, the category information of the candidate category grid cluster obtained by connected component analysis is a soil wall, which can better form the complete outline of the obstacle.

[0062] In one possible implementation, the candidate category grids are filtered to obtain a plurality of candidate category grid clusters; a minimum bounding rectangle of each candidate category grid cluster is generated, and a diagonal length of the minimum bounding rectangle is obtained; a candidate category grid cluster with a diagonal length not greater than a preset length threshold is determined as a first target grid cluster; and a candidate category grid cluster with a diagonal length greater than the preset length threshold is determined as a second target grid cluster.

[0063] Specifically, the island filtering refers to a process of aggregating the scattered candidate category grids into independent clusters through the connected domain analysis, which can eliminate isolated noise points and form the candidate obstacle region with actual physical meaning. After obtaining a plurality of candidate category grid clusters, a minimum bounding rectangle of each candidate category grid cluster can be generated, the minimum bounding rectangle refers to the minimum area rectangle that can completely enclose the candidate category grid cluster, and the diagonal length of the minimum bounding rectangle is obtained. According to the diagonal length, the candidate category grid cluster is distinguished.

[0064] Specifically, after the candidate category grid is filtered by the island filtering, a plurality of independent connected regions are formed, each independent connected region can be an obstacle, and different independent connected regions can be determined as a candidate category grid cluster. The diagonal length of the minimum bounding rectangle of the candidate category grid cluster is compared with the preset length threshold, the first target grid cluster and the second target grid cluster can be determined from the candidate category grid cluster, the preset length threshold can be preset, for example, set to 8 meters. When the diagonal length of the minimum bounding rectangle of the candidate category grid cluster does not exceed the preset length threshold, it can be considered that the grid cluster belongs to the small size obstacle category, and is determined as the first target grid cluster. When the diagonal length of the minimum bounding rectangle of the candidate category grid cluster exceeds the preset length threshold, it can be considered that the grid cluster belongs to the large size obstacle category, and is determined as the second target grid cluster. Through the island filtering of the candidate category grid, different sizes of obstacle types such as gravel, rockfall, and earth wall can be effectively distinguished.

[0065] S140: classifying the candidate category grid cluster based on the preset classification condition and the preset reference semantic map, and determining the first category obstacle and the second category obstacle, the preset classification condition is associated with the position information and the height value of the candidate category grid cluster, and the preset reference semantic map includes the surrounding area of the current position of the target device.

[0066] Specifically, through the preset classification condition, the candidate category grid cluster can be further classified. For example, when the category information of the candidate category grid cluster is an earth wall, the candidate category grid cluster can be classified into the first category obstacle and the second category obstacle through the preset classification condition. The first category obstacle can be an obstacle different from the category of the candidate category grid cluster, and the second category obstacle can be an obstacle same as the category of the candidate category grid cluster. For example, when the category information of the candidate category grid cluster is an earth wall, the first category obstacle can be gravel, mechanical parts, etc., and the second category obstacle is an earth wall. Here, no limitation is made. By distinguishing the first category obstacle and the second category obstacle, the obstacles in the surrounding area can be better distinguished.

[0067] Specifically, the reference semantic map can be constructed in various ways. For example, the reference semantic map can be constructed according to the point cloud data collected by the sensor in real time. The reference semantic map can also be a grid map, which can be used for comparison with the grids corresponding to the first category of obstacles and the second category of obstacles. The reference semantic map can include information such as the lane where the target device is located and the work area. It should be noted that the above description of the construction method of the reference semantic map is only exemplary, and other construction methods can exist in actual applications, which are not limited in particular.

[0068] Specifically, the preset classification condition can be classified according to the position information and height information of the candidate category grid cluster, to obtain the corresponding first category of obstacles and the second category of obstacles. The first category of obstacles can be gravel, and the second category of obstacles can be a soil wall. Figure 4 As shown in the figure, the orange area in the figure is the second category of obstacles displayed in the reference semantic map, and the blue area in the middle is the second category of obstacles obtained by the laser radar of the target device, that is, the second category of obstacles obtained by the point cloud data. The black box in the example in the figure is the first category of obstacles obtained after classification, that is, small objects such as gravel and sand on the lane.

[0069] In one possible implementation, the candidate category grid cluster is classified based on the preset classification condition and the preset reference semantic map to obtain the first category of obstacles and the second category of obstacles, including: if the first target grid cluster is located on one side of the lane in the reference semantic map, and there is a ground category grid within the preset distance range of the first target grid cluster, and the average height value of the first target grid cluster is greater than the second preset height value, the first target grid cluster is determined as the second category of obstacles, and the second category of obstacles is displayed in the reference semantic map; if the first target grid cluster is located on the lane in the reference semantic map, and there is no ground category grid within the preset distance range of the first target grid cluster, and the average height value of the first target grid cluster is greater than the third preset height value, the first target grid cluster is determined as the first category of obstacles, and the first category of obstacles is displayed in the reference semantic map; the second target grid cluster is determined as the second category of obstacles, and the second category of obstacles corresponding to the second target grid cluster is displayed in the reference semantic map.

[0070] In one possible implementation, the first category of obstacles is gravel, and the second category of obstacles is a soil wall.

[0071] Specifically, the first type of obstacle can be small objects such as gravel, sandstone, etc. The height value of the grid corresponding to the gravel is relatively low. When the small objects such as gravel, sandstone, etc. exist on the lane, if they cannot be identified, it is easy to cause danger to the driving of the vehicle. Therefore, the existence of small objects such as gravel, sandstone, etc. on the lane is needed as a classification condition. The second type of obstacle can be a soil wall. The height value of the grid corresponding to the soil wall is relatively high. The soil wall generally exists on both sides of the lane and has a large volume. The driving path of the vehicle is generally formulated to avoid the soil wall during formulation. Therefore, the soil wall on both sides of the lane can be used as a classification condition. The height value of the second target grid cluster is greater than the third preset height threshold value. The second target grid cluster can be directly determined as the second type of obstacle, i.e. the soil wall. The position corresponding to the second target grid cluster on the reference semantic map can be displayed.

[0072] Specifically, the first target grid cluster can be further classified according to the preset classification condition. The second preset height value can be preset. The second preset height value can be greater than the third preset height value. For example, the third preset height value is 0.5 m, and the second preset height value is between 0.8-1 m. Based on the position of the first target grid cluster in the reference semantic map, the first target grid cluster is classified as the first type of obstacle or the second type of obstacle. The preset distance range is used to determine whether there is a ground type grid cluster around the first target grid cluster. The preset distance range can be preset. For example, the preset distance range is 0.5 m. For example, there is a ground type grid around a certain first target grid cluster within 0.5 m, and the first target grid cluster is located on one side of the lane. The average height value is 0.8 m. It can be determined that the first target grid cluster is the first type of obstacle. The first type of obstacle, i.e. the sandstone, is displayed at the corresponding position in the reference semantic map. Based on the position correlation judgment of the first target grid, the ground grid existence detection, and the multiple rule screening of the average height value, the first target grid cluster and the second target grid cluster are further classified. The fixed facilities at the road edge and the temporary obstacles on the road surface can be accurately distinguished. For example, the sandstone and the soil wall can be distinguished. The safety of the target device in the work area can be improved.

[0073] Exemplarily, the target device can be a vehicle, first, the sensors on the vehicle collect data of the surrounding environment in real time, for example, point cloud data, radar data and image data, the point cloud data can be rasterized to obtain a raster result, each grid in the raster result includes category information and a height value, according to the category information of each grid, the grids are classified into first category grids and second category grids, the category information of the first category grids can be a vehicle or sand, after obtaining the first category grids, the third category obstacles corresponding to the first category grids can be directly displayed on the reference semantic map, the third category obstacles include vehicles and sand, and the category information of the second category grids can be gravel or a soil wall. After obtaining the second category grids, the second category grids can be subjected to a moving average processing to obtain an average height value of each second category grid, and the average height value is compared with a first preset height threshold to respectively obtain a candidate category grid and a ground category grid, the category information of the candidate category grid is a soil wall, and the category information of the ground category grid is ground, at this time, the grids corresponding to the gravel are classified in the candidate category grid. Then, the first candidate category grid is clustered, for example, the first candidate category grid is subjected to island filtering to obtain a plurality of candidate category grid clusters, and then the candidate category grid clusters are classified according to the diagonal line length of the minimum bounding rectangle of the candidate category grid clusters to obtain a first target grid cluster and a second target grid cluster, the second target grid cluster is a grid cluster corresponding to the soil wall, and the first target grid cluster can include a grid cluster corresponding to the soil wall or a grid cluster corresponding to the gravel. Then, the first target grid cluster is further subjected to more detailed classification according to a preset classification condition, according to the height value of the first target grid cluster, whether there is a ground category grid around, whether it is on the lane or on one side of the lane, to respectively obtain a first target grid cluster of a first category obstacle (gravel) and a first target grid cluster of a second category obstacle (soil wall), and the second target grid cluster can be directly confirmed as the second category obstacle (soil wall), and the corresponding obstacles are respectively displayed in the reference semantic map.

[0074] The obstacle classification method of the application comprises the following steps: obtaining a grid result corresponding to a surrounding area of a current position of a target device based on data collected by at least one sensor on the target device, each grid in the grid result being provided with category information and a height value; determining candidate category grids from the grid result according to the category information and the height value; clustering the candidate category grids to obtain a plurality of candidate category grid clusters, wherein each candidate category grid cluster comprises at least one candidate category grid; and classifying the candidate category grid clusters based on a preset classification condition and a preset reference semantic map to determine a first category obstacle and a second category obstacle, wherein the preset classification condition is associated with position information and height values of the candidate category grid clusters, the preset reference semantic map comprises the surrounding area of the current position of the target device, can process data collected by the sensor, and classifies obstacles in the surrounding area, thereby improving the accuracy of obstacle classification and the safety of vehicle driving.

[0075] Corresponding to the foregoing application function implementation method embodiments, the application further provides an obstacle classification device, an electronic device, and corresponding embodiments.

[0076] Figure 2 FIG. 1 is a structural schematic diagram of an obstacle classification device according to an embodiment of the application.

[0077] Referring to FIG. 1, Figure 2 An obstacle classification device 200 comprises:

[0078] An acquisition module 210 is configured to obtain a grid result corresponding to a surrounding area of a current position of a target device based on data collected by at least one sensor on the target device, each grid in the grid result being provided with category information and a height value.

[0079] A determination module 220 is configured to determine candidate category grids from the grid result according to the category information and the height value.

[0080] A clustering module 230 is configured to cluster the candidate category grids to obtain a plurality of candidate category grid clusters, wherein each candidate category grid cluster comprises at least one candidate category grid.

[0081] A classification module 240 is configured to classify the candidate category grid clusters based on a preset classification condition and a preset reference semantic map to obtain a first category obstacle and a second category obstacle, wherein the preset classification condition is associated with position information and height values of the candidate category grid clusters, and the preset reference semantic map comprises the surrounding area of the current position of the target device.

[0082] In a possible implementation, the acquisition module 210 is further configured to input data collected by the at least one sensor in real time into the multi-modal neural network model to obtain a grid result output by the multi-modal neural network model, where the data collected by the sensor in real time includes point cloud data, image data, and radar data of a surrounding area of a current position of the target device.

[0083] In a possible implementation, the determination module 220 is further configured to determine, from the grid result, a first category grid according to category information of each grid; and display, based on category information of the first category grid, a third category obstacle in the reference semantic map, where the third category obstacle includes a vehicle and sand.

[0084] In a possible implementation, the determination module 220 is further configured to determine, from the grid result, a second category grid according to category information of each grid; perform sliding average processing on an initial height value of the second category grid to obtain an average height value of the second category grid; determine, as a candidate category grid, a second category grid whose average height value is greater than a first preset height threshold; and determine, as a ground category grid, a second category grid whose average height value is not greater than the first preset height threshold.

[0085] In a possible implementation, the determination module 220 is further configured to include, in a formula for performing sliding average processing on the second category grid, a target weight.

[0086]

[0087] wherein, the target weight is positively correlated with a number of observations of a current grid, the average height value of the current grid, the initial height value of the current grid.

[0088] In a possible implementation, the clustering module 230 is further configured to perform island filtering on the candidate category grid to obtain a plurality of candidate category grid clusters; generate a minimum bounding rectangle of each candidate category grid cluster and obtain a diagonal length of the minimum bounding rectangle; determine, as a first target grid cluster, a candidate category grid cluster whose diagonal length is not greater than a preset length threshold; and determine, as a second target grid cluster, a candidate category grid cluster whose diagonal length is greater than the preset length threshold.

[0089] In a possible implementation, the classification module 240 is further configured to: if the first target grid cluster is located on one side of a lane in the reference semantic map, and there is a ground category grid within a preset distance range of the first target grid cluster, and the average height value of the first target grid cluster is greater than a second preset height value, determine the first target grid cluster as a first category obstacle, and display the first category obstacle in the reference semantic map; if the first target grid cluster is located on the lane in the reference semantic map, and there is no ground category grid within the preset distance range of the first target grid cluster, and the average height value of the first target grid cluster is greater than a first preset height value, determine the first target grid cluster as a second category obstacle, and display the second category obstacle in the reference semantic map; and determine the second target grid cluster as the second category obstacle, and display the first category obstacle in the reference semantic map.

[0090] The obstacle classification device provided in the application comprises an acquisition module, a determination module, a clustering module, and a classification module. The acquisition module is configured to obtain a grid result corresponding to a surrounding area of a current position of a target device based on data collected by at least one sensor on the target device, and each grid in the grid result has category information and a height value. The determination module is configured to determine candidate category grids from the grid result according to the category information and the height value. The clustering module is configured to cluster the candidate category grids to obtain a plurality of candidate category grid clusters, wherein each candidate category grid cluster includes at least one candidate category grid. The classification module is configured to classify the candidate category grid clusters based on a preset classification condition and a preset reference semantic map to obtain a first category obstacle and a second category obstacle. The preset classification condition is associated with position information and height values of the candidate grid clusters. The preset reference semantic map includes the surrounding area of the current position of the target device, can process data obtained by the sensor, and can classify obstacles in the surrounding area, thereby improving the accuracy of obstacle classification and the safety of vehicle driving.

[0091] As to the apparatus in the above-mentioned embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0092] The application further provides an electronic device. Figure 3 Fig. 1 is a schematic diagram of a hardware structure of an embodiment of the electronic device. The electronic device comprises a memory 320 and at least one processor 310. The memory 320 is electrically connected to the at least one processor 310. The memory 320 stores instructions. The at least one processor 310 invokes the instructions in the memory 320, so that the electronic device performs the obstacle classification method according to any one of the preceding embodiments of the application.

[0093] In particular, the processor 310 described above can include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or can be configured as one or more integrated circuits to implement the embodiments of the present application.

[0094] The memory 320 can include mass storage for data or instructions. By way of example, and not limitation, the memory 320 can include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. The memory 320 can include removable or non-removable (or fixed) media, where appropriate. The memory 320 can be considered a computer-readable medium, where appropriate. The memory 320 can be internal or external to the integrated gateway disaster recovery device, where appropriate. In particular embodiments, the memory 320 is nonvolatile, solid-state memory. In particular embodiments, the memory 320 includes read-only memory (ROM). Where appropriate, this ROM can be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these. The memory 320 can be a computer-readable medium that carries computer code for the integrated gateway disaster recovery device (e.g., within a

[0095] In one example, the control device also includes a communication interface 330 and a bus 340. The processor 310, the memory 320, and the communication interface 330 are connected through the bus 340 and accomplish the communication among each other.

[0096] The communication interface 330 is mainly used to realize the communication between the modules, devices, units and / or equipment in the embodiments of the present application.

[0097] The bus 340 includes hardware, software, or both, that couples components of the online data traffic billing device to each other in a known manner. By way of example, and not limitation, the bus can include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or interconnect, or a combination of two or more of these. Where appropriate, the bus 340 can include one or more buses. Although the present application describes and illustrates a particular bus, this application contemplates any suitable bus or interconnect.

[0098] In addition, in combination with the obstacle classification method in the above-mentioned embodiments, the embodiments of the present application can provide a computer readable storage medium for implementation. The computer readable storage medium has instructions stored thereon, and the instructions are executed by a processor to implement any of the obstacle classification methods in the above-mentioned embodiments.

[0099] The present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of well-known methods are omitted herein. In the above-mentioned embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order of the steps, after understanding the spirit of the present application.

[0100] The functional blocks shown in the above structural block diagrams can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are program or code segments used to perform the required tasks. The program or code segments can be stored in a machine readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. The "machine readable medium" can include any medium capable of storing or transmitting information. Examples of the machine readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segments can be downloaded via a computer network such as the Internet, an intranet, etc.

[0101] It should also be noted that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above-mentioned steps, that is, the steps can be performed in the order mentioned in the embodiments, or in an order different from the embodiments, or several steps can be performed simultaneously.

[0102] Alternatively, the present application also provides a computer program product capable of implementing part or all of each step of the above-mentioned embodiments, and the computer program product includes a computer program / instruction, which, when executed by a processor, implements part or all of each step of the above-mentioned embodiments.

[0103] The above is merely a specific implementation of the present application. As can be clearly understood by a person skilled in the art from the above description, for the convenience and brevity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be described herein again. It should be understood that the protection scope of the present application is not limited in this way, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed in the present application, and these modifications or replacements should be covered in the protection scope of the present application.

Claims

1. An obstacle classification method, characterized in that, include: Based on data collected in real time by at least one sensor on the target device, a grid result corresponding to the surrounding area of ​​the current position of the target device is obtained, wherein each grid cell in the grid result carries category information and height value; Based on the category information and the height value, candidate category rasters are determined from the raster results; Clustering the candidate category rasters yields several candidate category raster clusters, each containing at least one candidate category raster. This includes: performing island filtering on the candidate category rasters to obtain several candidate category raster clusters; generating the minimum bounding rectangle for each candidate category raster cluster and obtaining the diagonal length of the minimum bounding rectangle; identifying candidate category raster clusters whose diagonal length is not greater than a preset length threshold as first target raster clusters; and identifying candidate category raster clusters whose diagonal length is greater than the preset length threshold as second target raster clusters. Based on preset classification conditions and a preset reference semantic map, the candidate category grid clusters are classified to determine a first category obstacle and a second category obstacle. The preset classification conditions are associated with the location information and height value of the candidate category grid clusters. The preset reference semantic map includes the area surrounding the current location of the target device, including: if the first target grid cluster is located on one side of a lane in the reference semantic map, and there are ground category grids within a preset distance range of the first target grid cluster, and the average height value of the first target grid cluster is greater than a second preset height value, then the first target grid cluster is determined to be a second category obstacle and displayed in the reference semantic map; if the first target grid cluster is located on a lane in the reference semantic map, and there are no ground category grids within a preset distance range of the first target grid cluster, and the average height value of the first target grid cluster is greater than a third preset height value, then the first target grid cluster is determined to be a first category obstacle and displayed in the reference semantic map; the second target grid cluster is determined to be a second category obstacle, and the second category obstacle corresponding to the second target grid cluster is displayed in the reference semantic map.

2. The method according to claim 1, characterized in that, The process of obtaining a grid result corresponding to the area surrounding the current location of the target device based on data collected in real time by at least one sensor on the target device includes: Data collected in real time by at least one sensor is input into a multimodal neural network model to obtain the grid result output by the multimodal neural network model; wherein, the data collected in real time by the sensor includes: point cloud data, image data and radar data of the area surrounding the current location of the target device.

3. The method according to claim 1, characterized in that, Also includes: Based on the category information of each of the graticles, a first category of graticles is determined from the graticle results; Based on the category information of the first category grid, a third category of obstacles is displayed in the reference semantic map; wherein, the third category of obstacles includes: vehicles and sandstorms.

4. The method according to claim 1, characterized in that, The step of determining candidate category rasters from the raster results based on the category information and the height value includes: Based on the category information of each raster, determine the second category raster from the raster results; The initial height values ​​of the second category of grid cells are averaged by performing a moving average process to obtain the average height value of the second category of grid cells. The second category grid with an average height value greater than a first preset height threshold is determined as the candidate category grid; The second category grid with an average height value not greater than the first preset height threshold is determined as the ground category grid.

5. The method according to claim 4, characterized in that, The step of performing a moving average process on the height values ​​of the second category of grid cells to obtain the average height value of the second category of grid cells includes: The formula for applying a moving average to the second category of graticles includes: in, The target weight is positively correlated with the number of observations of the current grid. This represents the average height of the current grid cell. This is the initial height value of the current grid.

6. The method according to claim 1, characterized in that, include: The first category of obstacles is gravel, and the second category of obstacles is earthen walls.

7. An obstacle classification device, characterized in that, include: The acquisition module is used to obtain a grid result corresponding to the surrounding area of ​​the current position of the target device based on data collected in real time by at least one sensor on the target device. Each grid in the grid result carries category information and height value. A determination module is used to determine candidate category rasters from the raster results based on the category information and the height value; A clustering module is used to cluster the candidate category rasters to obtain several candidate category raster clusters, wherein each candidate category raster cluster includes at least one candidate category raster; the clustering module includes: performing island filtering on the candidate category rasters to obtain several candidate category raster clusters; generating the minimum bounding rectangle of each candidate category raster cluster and obtaining the diagonal length of the minimum bounding rectangle; determining the candidate category raster clusters whose diagonal length is not greater than a preset length threshold as the first target raster cluster; and determining the candidate category raster clusters whose diagonal length is greater than the preset length threshold as the second target raster cluster. A classification module is used to classify the candidate category grid clusters based on preset classification conditions and a preset reference semantic map to obtain a first category obstacle and a second category obstacle. The preset classification conditions are associated with the location information and height value of the candidate category grid clusters. The preset reference semantic map includes the area surrounding the current location of the target device, including: if the first target grid cluster is located on one side of a lane in the reference semantic map, and there are ground category grids within a preset distance range of the first target grid cluster, and the average height value of the first target grid cluster is greater than a second preset height value, then the first target grid cluster is confirmed as an obstacle. If the first target grid cluster is located on a lane in the reference semantic map, and there are no ground-type grids within a preset distance range of the first target grid cluster, and the average height value of the first target grid cluster is greater than a third preset height value, then the first target grid cluster is determined as the first type of obstacle and displayed in the reference semantic map; the second target grid cluster is determined as the second type of obstacle, and the second type of obstacle corresponding to the second target grid cluster is displayed in the reference semantic map.

8. An electronic device, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-6.

Images