Obstacle classification method and related device

Abstract

The invention relates to an obstacle classification method and a related device thereof, and the method comprises the steps: obtaining a grid result corresponding to a surrounding region of a current position of target equipment based on data collected by at least one sensor on the target equipment in real time, and each grid in the grid result carries category information and a height value; determining candidate category grids from the grid result according to the category information and the height value; clustering the candidate category grids to obtain a plurality of candidate category grid clusters; and based on a preset classification condition and a preset reference semantic map, the candidate category grid clusters are classified, a first category barrier and a second category barrier are determined, and the preset classification condition is associated with the position information and the height value of the candidate category grid clusters. According to the scheme provided by the invention, the data acquired by the sensor can be processed, the obstacles in the surrounding area are classified, and the accuracy of obstacle classification and the safety of vehicle driving are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to an obstacle classification method and a related device thereof. BACKGROUND

[0002] The open-pit mine road environment is complex, often accompanied by mine obstacles such as stone piles, soil slopes, deep pits, retaining walls, pedestrians, and auxiliary operation vehicles. In order to prevent mine cars driving on the mine road from colliding with mine obstacles, mine obstacle detection needs to be performed on the mine road where the mine cars are driving.

[0003] In related technologies, data obtained by a single sensor is generally used to classify obstacles. However, in a mine environment, unknown types of obstacles such as ore, gravel, and small parts may exist. The existing detection method is difficult to detect different types of obstacles, resulting in low obstacle detection accuracy and difficulty in ensuring the driving safety of vehicles. SUMMARY

[0004] To solve or partially solve the problems in related technologies, the present application provides an obstacle classification method and a related device thereof, which can process data obtained by a sensor, classify obstacles in a surrounding area, and improve the accuracy of obstacle classification and the safety of vehicle driving.

[0005] The first aspect of the present application provides an obstacle classification method, including obtaining a grid result corresponding to a surrounding area of a current position of a target device based on data collected in real time by at least one sensor on the target device, each grid in the grid result having category information and a height value; determining a candidate category grid from the grid result according to the category information and the height value; clustering the candidate category grid to obtain a plurality of candidate category grid clusters, wherein the candidate category grid clusters include at least one candidate category grid; classifying the candidate category grid clusters based on a preset classification condition and a preset reference semantic map to determine a first category obstacle and a second category obstacle, the preset classification condition being associated with position information and a height value of the candidate category grid cluster, and the preset reference semantic map including the surrounding area of the current position of the target device.

[0006] In combination with the first aspect, in a possible implementation manner of the first aspect, the obtaining of the grid result corresponding to the surrounding area of the current position of the target device based on the data collected in real time by the at least one sensor on the target device includes: inputting the data collected in real time by the at least one sensor into a multi-modal neural network model to obtain the grid result output by the multi-modal neural network model; and the data collected in real time by the sensor includes point cloud data, image data, and radar data of the surrounding area of the current position of the target device.

[0007] With reference to the first aspect, in a possible implementation form of the first aspect, the method further includes: determining a first category grid from the grid results according to the category information of each grid; and displaying a third category obstacle in the reference semantic map based on the category information of the first category grid; wherein the third category obstacle includes a vehicle and a sandstorm.

[0008] With reference to the first aspect, in a possible implementation form of the first aspect, the determining the candidate category grid from the grid results according to the category information and the height value includes: determining a second category grid from the grid results according to the category information of each grid; performing a sliding average processing on an initial height value of the second category grid to obtain an average height value of the second category grid; determining the second category grid with the average height value greater than a first preset height threshold as the candidate category grid; and determining the second category grid with the average height value not greater than the first preset height threshold as a ground category grid.

[0009] With reference to the first aspect, in a possible implementation form of the first aspect, the performing the sliding average processing on the height value of the second category grid to obtain the average height value of the second category grid includes: The formula of the sliding average processing on the second category grid includes:

[0010] wherein, the target weight is positively correlated with the number of observations of the current grid, is the average height value of the current grid, is the initial height value of the current grid.

[0011] With reference to the first aspect, in a possible implementation form of the first aspect, the clustering the candidate category grid to obtain a plurality of candidate category grid clusters includes: performing island filtering on the candidate category grid to obtain a plurality of candidate category grid clusters; generating a minimum bounding rectangle of each candidate category grid cluster and obtaining a diagonal length of the minimum bounding rectangle; determining a first target grid cluster from the candidate category grid clusters with the diagonal length not greater than a preset length threshold; and determining a second target grid cluster from the candidate category grid clusters with the diagonal length greater than the preset length threshold.

[0012] ​In a possible implementation of the first aspect, the classifying the candidate category grid cluster based on the preset classification condition and the preset reference semantic map includes: if the first target grid cluster is located on one side of a lane in the reference semantic map, and the ground category grid exists within a preset distance range of the first target grid cluster, and an average height value of the first target grid cluster is greater than a second preset height value, the first target grid cluster is determined as the second category obstacle, and the second category obstacle is displayed in the reference semantic map; if the first target grid cluster is located on the lane in the reference semantic map, and the ground category grid does not exist within the preset distance range of the first target grid cluster, and the average height value of the first target grid cluster is greater than a third preset height value, the first target grid cluster is determined as the first category obstacle, and the first category obstacle is displayed in the reference semantic map; and the second target grid cluster is determined as the second category obstacle, and the second target grid cluster corresponding to the second category obstacle is displayed in the reference semantic map.

[0013] In a possible implementation of the first aspect, the first category obstacle is gravel, and the second category obstacle is a soil wall.

[0014] The second aspect of the present application provides an obstacle classification device, including: an acquisition module configured to obtain a grid result corresponding to a surrounding area of a current position of a target device based on data collected by at least one sensor on the target device in real time, each grid in the grid result being provided with category information and a height value; a determination module configured to determine candidate category grids from the grid result according to the category information and the height value; a clustering module configured to cluster the candidate category grids to obtain a plurality of candidate category grid clusters, wherein each candidate category grid cluster includes at least one candidate category grid; and a classification module configured to classify the candidate category grid clusters based on a preset classification condition and a preset reference semantic map to obtain a first category obstacle and a second category obstacle, the preset classification condition being associated with position information and a height value of the candidate category grid cluster, and the preset reference semantic map including the surrounding area of the current position of the target device.

[0015] The third aspect of the present application provides an electronic device, including: a processor; and a memory having executable code stored thereon, the executable code, when executed by the processor, causing the processor to perform the method described above.

[0016] The fourth aspect of the present application provides a computer readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0017] The technical solution provided by the present application can include the following beneficial effects: The obstacle classification method and the related device provided by the present application include: obtaining a grid result corresponding to a surrounding area of a current position of a target device based on data collected by at least one sensor on the target device, each grid in the grid result having category information and a height value; determining a candidate category grid from the grid result according to the category information and the height value; clustering the candidate category grid to obtain a plurality of candidate category grid clusters, wherein each candidate category grid cluster includes at least one candidate category grid; and classifying the candidate category grid cluster based on a preset classification condition and a preset reference semantic map to determine a first category obstacle and a second category obstacle, wherein the preset classification condition is associated with position information and a height value of the candidate category grid cluster, the preset reference semantic map includes the surrounding area of the current position of the target device, can process data obtained by the sensor, and can classify obstacles in the surrounding area, thereby improving the accuracy of obstacle classification and the safety of vehicle driving.

[0018] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present application. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings, in which like reference characters refer to like parts throughout the several views, and in which:

[0020] Figure 1 is a flowchart of an obstacle classification method shown in an embodiment of the present application; Figure 2 is a structural diagram of an obstacle classification device shown in an embodiment of the present application; Figure 3 is a structural diagram of an electronic device shown in an embodiment of the present application; Figure 4 is a reference semantic map including a first category obstacle and a second category obstacle shown in an embodiment of the present application; Figure 5 is a diagram of single-frame data collected by a target device shown in an embodiment of the present application; Figure 6 is a superimposed diagram of a reference semantic map and a diagram of single-frame data collected by a target device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0021] Embodiments of the present application will be described in more detail with reference to the drawings. Although the embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0022] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in this application and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0023] It should be understood that although the terms "first", "second", "third", etc. can be used in this application to describe various information, these information should not be limited by these terms. These terms are only used to distinguish the same type of information from each other. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information without departing from the scope of the present application. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0024] In a complex operation environment such as a mine, there are often a large number of obstacles, and the mine environment has the characteristics of complex terrain, various types of obstacles and dynamic changes, such as different vehicles, sand and the like. On the driving path of the vehicle, there are often obstacles such as gravel, sand, earth walls and the like. The traditional obstacle detection method can detect the earth wall on the driving path and display it on the map to let the vehicle avoid the earth wall. However, due to the low height and small volume of small objects such as sand and gravel, these small objects are easily detected as the ground during detection, and this type of obstacle is not displayed on the map. Especially when sand and gravel exist on the driving path of the vehicle, the vehicle cannot detect this type of obstacle, which makes the vehicle unable to avoid small objects such as sand and gravel during driving, causing danger to the driving of the vehicle.

[0025] To solve the above problems, the embodiments of the present application provide an obstacle classification method and related device, which can process the data obtained by the sensor and classify the obstacles on the surrounding area, thereby improving the accuracy of obstacle classification and the safety of vehicle driving.

[0026] The technical solutions of the embodiments of the present application will be described in detail below with reference to the drawings.

[0027] Figure 1 It is a flowchart of the obstacle classification method shown in an embodiment of the present application.

[0028] See also Figure 1 and Figures 4-6 , an obstacle classification method, comprising: S110: Based on data collected in real time by at least one sensor on the target device, a grid result corresponding to the surrounding area of ​​the current position of the target device is obtained, where each grid in the grid result carries category information and a height value.

[0029] Specifically, the target device may be a vehicle, in which case its current operating environment may be a mining environment. The target device may include multiple sensors, each of which may be multi-source sensors. The data collected in real time by these sensors may be multi-source data. For example, the at least one sensor may include a lidar for collecting lidar data in real time, and may also include a visual sensor for collecting visual data in real time. As another example, the visual sensor may be a camera. The at least one sensor may also include other specific implementations, which are not limited to these specific implementations.

[0030] Specifically, the raster result refers to the process of converting three-dimensional point cloud data into a two-dimensional raster matrix. Each grid records the category information and corresponding height value of the obstacles in the area. The raster result can be a raster map with a resolution of 0.25. For example, when the category information of the grid is vehicle, it can be determined that there is a vehicle in the real area corresponding to the grid.

[0031] Exemplarily, the surrounding area of ​​the current location of the target device can be an area within a preset range in front of and behind the current location of the target device. For example, the preset range can be 100 meters or 200 meters, or other values. For example, the surrounding area is an area within 100 meters in front of and 100 meters behind the current location of the target device. Therefore, there is no limitation on the specific implementation of the surrounding area.

[0032] In one possible implementation, based on data collected in real time by at least one sensor on the target device, a grid result corresponding to the surrounding area of ​​the current location of the target device is obtained, including: inputting the data collected in real time by at least one sensor into a multimodal neural network model to obtain a grid result output by the multimodal neural network model; wherein the data collected in real time by the sensor includes: point cloud data, image data and radar data of the surrounding area of ​​the current location of the target device, and the target device can be a movable device, such as a mining truck, an excavator, a bulldozer, a small vehicle or other types of vehicles, and can also be a drone or a robot, etc., without specific limitation.

[0033] Specifically, the multi-modal neural network model can be a multi-task multi-sensor fusion model with unified bird's-eye view representation (BEVFusion) model, which can fuse multi-modal sensor data (such as cameras and lidar) through a unified bird's-eye view (BEV) space, support multi-task perception (such as 3D target detection, BEV map segmentation, etc.). The target neural network model can also be other types of neural network models, for example, a neural network model including a convolutional neural network and a point cloud processing network fusion, which is not limited in particular.

[0034] Specifically, the sensors on the target device include multiple sensors, through which point cloud data, image data and radar data can be obtained. The data collected by the multiple sensors in real time can be subjected to feature extraction and feature fusion in the multi-modal neural network model. After the collected data is subjected to feature extraction and feature fusion, the perception result of the surrounding area of the current position of the target device output by the multi-modal neural network model is obtained, i.e., the above-mentioned grid result.

[0035] Exemplarily, the point cloud data, image data and radar data collected by the sensors in real time can be synchronously input into the multi-modal neural network model for processing. The multi-modal neural network model first extracts the features of each modality through a dedicated branch, for example, uses a PointNet++ network to process point cloud data, a ResNet to process image data, and a one-dimensional convolutional network to process radar data, and then aligns and splices the multi-modal features through a feature fusion module to generate a comprehensive feature representation containing spatial, texture and dynamic information. The comprehensive feature passes through a decoder module to generate a grid result, each grid containing class information and height value. The point cloud data can ensure spatial positioning accuracy, the image data can enhance the accuracy of class recognition, and the radar data can provide basic perception ability in the mine environment. Through multi-modal fusion of the point cloud data, image data and radar data, environmental interference factors can be effectively excluded, which is particularly suitable for mine scenes with a large amount of dust, vibration and complex terrain, can avoid misjudgment of the category of obstacles due to failure of a single sensor, and the complementary features of three-dimensional point cloud and two-dimensional image can significantly reduce the probability of misjudging the ground concave-convex as an obstacle.

[0036] S120: determining a candidate category grid from the grid result according to the class information and the height value.

[0037] Specifically, the category information of the grid can include a vehicle, an earth wall, sand and the like, and the grid further includes a height value. The candidate category grid can be a potential obstacle region filtered by the height threshold. For example, when the category information of the grid is a vehicle and the height value is greater than the height threshold, the grid can be determined as the candidate category grid. Here, no limitation is made. By filtering the candidate category grid, the subsequent classification of the grid corresponding to the obstacle can be better performed.

[0038] In a possible implementation, the method further includes: determining, according to the category information of each grid, a first category grid from the grid result; and displaying, based on the category information of the first category grid, a third category obstacle in the reference semantic map; wherein the third category obstacle includes a vehicle and sand.

[0039] Specifically, the first category grid refers to a grid whose category information corresponds to a vehicle or sand in the grid result. The first category grid can be clustered to obtain a grid cluster aggregated by the first category grids of a unified category, and the grid cluster is determined as the third category obstacle. The boundary coordinates of the grid cluster are mapped to a corresponding position of the reference semantic map. For example, the grid cluster corresponding to the vehicle is labeled as a vehicle on the reference semantic map, and the grid cluster corresponding to the sand is labeled as a dynamic environmental interference. The reference semantic map displays the positions and distribution ranges of the vehicle and the sand by superimposing real-time obstacle information. By directly extracting the category information of the corresponding grid of the vehicle and the sand and associating the reference semantic map, the vehicle and the sand and other dynamic obstacles can be effectively distinguished in the mine scene, and the working safety of the target device in the mine environment is improved.

[0040] In a possible implementation, the method further includes: determining, according to the category information of each grid, a second category grid from the grid result; performing a sliding average processing on an initial height value of the second category grid to obtain an average height value of the second category grid; determining a second category grid with an average height value greater than a first preset height threshold as a candidate category grid; and determining a second category grid with an average height value not greater than the first preset height threshold as a ground category grid.

[0041] Specifically, the second category grid refers to a grid preliminarily classified as an earth wall category and a ground category by the multi-modal model. The initial height value of the second category grid is subjected to the sliding average processing to obtain the average height value of the second category grid. The sliding average processing refers to a weighted average calculation method based on a time sequence, which can be used to eliminate single-frame data noise interference. The average height value can be better used for subsequent classification of the second category grid.

[0042] Specifically, the first preset height threshold can be a critical height value for the ground and the obstacle, the first preset height threshold can be dynamically set according to the terrain undulation characteristics of the mine environment, for example, can be set to 0.3-0.5 meters, the second type of grid higher than the first preset height threshold can be determined as the candidate type of grid, at this time, the type of grid can be a soil wall, the second type of grid not greater than the first preset height threshold can be determined as the ground type of grid, the ground type of grid can be superimposed on the reference semantic map in real time, and the position and distribution range of the ground on the reference semantic map can be displayed, the height of the obstacle can be dynamically distinguished, and the robustness of the obstacle detection in the mine scene is improved.

[0043] In a possible implementation, the height value of the second type of grid is subjected to a sliding average processing to obtain an average height value of the second type of grid, including: The formula for the sliding average processing of the second type of grid includes:

[0044] wherein, The target weight is positively correlated with the observation number of the current grid, The average height value of the current grid, The initial height value of the current grid.

[0045] Specifically, the target weight refers to a coefficient associated with the observation number, which can specifically map the observation number to a weight value in the range of 0-1 through a linear or nonlinear function, for example, the weight value increases by 0.05 each time the observation number increases by one, and the average height value is obtained through accumulation of multiple frames of data.

[0046] Specifically, in the mine environment, the initial height value collected by the sensor can have abnormal fluctuations due to noise or instantaneous shielding, the initial height value of the current frame is superimposed with the historical average height value according to the weight, and the weight coefficient is dynamically adjusted according to the number of times the grid is observed. For example, when a certain grid is detected for the first time, the observation number is 1, and the weight value can be set to 0.2, and as the observation number increases to 5 times, the weight value gradually increases to 0.5. The average height value of the grid obtained through the sliding average calculation is more accurate, can effectively eliminate the instantaneous noise interference in a single frame of data, solves the problem of unstable height measurement in the mine scene due to dust, equipment vibration and other factors, and avoids false detection of obstacles due to height misjudgment.

[0047] S130: clustering the candidate type of grid to obtain a plurality of candidate type of grid clusters, wherein the candidate type of grid cluster includes at least one candidate type of grid.

[0048] Specifically, after obtaining a plurality of candidate category grids, the candidate category grids can be clustered to obtain candidate category grid clusters, which can be obstacle aggregation areas formed by connected component analysis. When forming the candidate category grid clusters, the candidate category grid clusters can be pre-provided with category information. For example, when the category information of the candidate category grid is a soil wall, the category information of the candidate category grid cluster obtained by connected component analysis is a soil wall, which can better form the complete outline of the obstacle.

[0049] In a possible implementation, the candidate category grids are subjected to island filtering to obtain a plurality of candidate category grid clusters; a minimum bounding rectangle of each candidate category grid cluster is generated, and a diagonal length of the minimum bounding rectangle is obtained; a candidate category grid cluster with a diagonal length not greater than a preset length threshold is determined as a first target grid cluster; and a candidate category grid cluster with a diagonal length greater than the preset length threshold is determined as a second target grid cluster.

[0050] Specifically, island filtering refers to a process of aggregating scattered candidate category grids into independent clusters through connected component analysis, which can eliminate isolated noise points and form obstacle candidate areas with actual physical significance. After obtaining a plurality of candidate category grid clusters, a minimum bounding rectangle of each candidate category grid cluster can be generated, which is a minimum area rectangle that can completely enclose the candidate category grid cluster, and a diagonal length of the minimum bounding rectangle is obtained. According to the diagonal length, the candidate category grid clusters can be distinguished.

[0051] Specifically, after the candidate category grids are subjected to island filtering, a plurality of independent connected regions are formed, each of which can be an obstacle. Different independent connected regions can be determined as candidate category grid clusters. The diagonal length of the minimum bounding rectangle of the candidate category grid cluster can be compared with a preset length threshold, and the first target grid cluster and the second target grid cluster can be determined from the candidate category grid cluster. The preset length threshold can be pre-set, for example, to 8 meters. When the diagonal length of the minimum bounding rectangle of the candidate category grid cluster does not exceed the preset length threshold, it can be considered that the grid cluster belongs to a small-size obstacle category, and is determined as the first target grid cluster. When the diagonal length of the minimum bounding rectangle of the candidate category grid cluster exceeds the preset length threshold, it can be considered that the grid cluster belongs to a large-size obstacle category, and is determined as the second target grid cluster. By subjecting the candidate category grids to island filtering, different sizes of obstacle types, such as gravel, rockfall, and soil wall, can be effectively distinguished.

[0052] S140: classify the candidate category grid cluster based on a preset classification condition and a preset reference semantic map, determine the first category obstacle and the second category obstacle, the preset classification condition is associated with the position information and the height value of the candidate category grid cluster, and the preset reference semantic map includes a surrounding area of a current position of the target device.

[0053] Specifically, through the preset classification condition, the candidate category grid cluster can be further classified. For example, when the category information of the candidate category grid cluster is a soil wall, the candidate category grid cluster can be classified into the first category obstacle and the second category obstacle through the preset classification condition. The first category obstacle can be an obstacle different from the category of the candidate category grid cluster, and the second category obstacle can be an obstacle same as the category of the candidate category grid cluster. For example, when the category information of the candidate category grid cluster is a soil wall, the first category obstacle can be gravel, mechanical parts, etc., and the second category obstacle is a soil wall. Here, no limitation is made. By distinguishing the first category obstacle and the second category obstacle, the obstacles in the surrounding area can be better distinguished.

[0054] Specifically, the reference semantic map can be constructed in various ways. For example, the reference semantic map can be constructed according to the point cloud data collected by the sensor in real time. The reference semantic map can also be a grid map, which can be used to compare the grids corresponding to the first category obstacle and the second category obstacle. The reference semantic map can include information such as a lane and a work area where the target device is located. It should be noted that the above description of the construction method of the reference semantic map is only exemplary, and other construction methods can exist in actual applications, which are not limited herein. When classifying the candidate category grid cluster, the first category obstacle and the second category obstacle can be determined based on the position information of the candidate category cluster in the reference semantic map. After determining the first category obstacle and the second category obstacle, the obstacles of this category can be marked in the reference semantic map, which can more accurately classify the obstacles and improve the driving safety of the target device.

[0055] Specifically, the preset classification condition can be used to classify the candidate category grid cluster based on the position information and the height information of the candidate category grid cluster, to obtain the corresponding first category obstacle and the second category obstacle. The first category obstacle can be gravel, and the second category obstacle can be a soil wall. Figure 4 As shown in FIG. 8, the orange area in the figure is the second category obstacle displayed in the reference semantic map, and the blue area in the middle is the second category obstacle obtained by the laser radar of the target device, i.e., the second category obstacle obtained by the point cloud data. The black box in the figure is the first category obstacle obtained after classification, i.e., small objects such as gravel and sand on the lane.

[0056] In a possible implementation, the first-class obstacle and the second-class obstacle are obtained by classifying the candidate-class grid cluster based on the preset classification condition and the preset reference semantic map, including: if the first target grid cluster is located on one side of a lane in the reference semantic map, and there is a ground-class grid within a preset distance range of the first target grid cluster, and an average height value of the first target grid cluster is greater than a second preset height value, the first target grid cluster is determined as the second-class obstacle, and the second-class obstacle is displayed in the reference semantic map; if the first target grid cluster is located on the lane in the reference semantic map, and there is no ground-class grid within the preset distance range of the first target grid cluster, and the average height value of the first target grid cluster is greater than a third preset height value, the first target grid cluster is determined as the first-class obstacle, and the first-class obstacle is displayed in the reference semantic map; the second target grid cluster is determined as the second-class obstacle, and the second-class obstacle corresponding to the second target grid cluster is displayed in the reference semantic map.

[0057] In a possible implementation, the first-class obstacle is gravel, and the second-class obstacle is a soil wall.

[0058] Specifically, the first-class obstacle can be small objects such as gravel and sandstone, and the height value of the grid corresponding to the gravel is relatively low. When the small objects such as gravel and sandstone exist on the lane, if they cannot be identified, they are likely to cause danger to the driving of the vehicle. Therefore, the existence of the small objects such as gravel and sandstone on the lane is used as a classification condition. The second-class obstacle can be a soil wall, and the height value of the grid corresponding to the soil wall is relatively high. The soil wall generally exists on both sides of the lane and has a large volume. In the process of formulating the driving path of the vehicle, the soil wall is generally avoided to formulate the driving path. Therefore, the soil wall on both sides of the lane can be used as a classification condition. The height value of the second target grid cluster is greater than the third preset height threshold value, and the second target grid cluster can be directly determined as the second-class obstacle, that is, the soil wall. The position corresponding to the second target grid cluster in the reference semantic map can be displayed.

[0059] Specifically, the first target grid cluster can be classified according to a preset classification condition, a second preset height value can be preset, the second preset height value can be greater than a third preset height value, for example, the third preset height value is 0.5 m, and the second preset height value is between 0.8-1 m, and then the first target grid cluster is classified as a first category obstacle or a second category obstacle based on a position of the first target grid cluster in the reference semantic map, a preset distance range is used to determine whether there is a ground category grid cluster around the first target grid cluster, the preset distance range can be preset, for example, the preset distance range is 0.5 m, for example, there is a ground category grid around a certain first target grid cluster within 0.5 m, and the first target grid cluster is located on one side of the lane, and the average height value is 0.8 m, so it can be determined that the first target grid cluster is a first category obstacle, and the first category obstacle, i.e., the sandstone, is displayed at the corresponding position in the reference semantic map. Based on the multiple rules of the position relevance judgment of the first target grid, the ground grid existence detection, and the average height value screening, the first target grid cluster and the second target grid cluster are further classified, which can accurately distinguish the fixed facilities on the road edge from the temporary obstacles on the road surface, for example, the sandstone can be distinguished from the earth wall, and the safety of the target device in the operation area can be improved.

[0060] Exemplarily, the target device can be a vehicle, first, the sensors on the vehicle collect data of the surrounding environment in real time, for example, point cloud data, radar data and image data, the point cloud data can be rasterized to obtain a raster result, each grid in the raster result includes category information and a height value, according to the category information of each grid, the grids are classified into first category grids and second category grids, the category information of the first category grids can be a vehicle or sand, after obtaining the first category grids, the third category obstacles corresponding to the first category grids can be directly displayed on the reference semantic map, the third category obstacles include vehicles and sand, and the category information of the second category grids can be gravel or a soil wall. After obtaining the second category grids, the second category grids can be subjected to a sliding average processing to obtain an average height value of each second category grid, and the average height value is compared with a first preset height threshold to respectively obtain a candidate category grid and a ground category grid, the category information of the candidate category grid is a soil wall, and the category information of the ground category grid is ground, at this time, the grids corresponding to the gravel are classified in the candidate category grid. Then, the first candidate category grid is clustered, for example, the first candidate category grid is subjected to island filtering to obtain a plurality of candidate category grid clusters, and then the candidate category grid clusters are classified according to the diagonal line length of the minimum bounding rectangle of the candidate category grid clusters to obtain a first target grid cluster and a second target grid cluster, the second target grid cluster is a grid cluster corresponding to the soil wall, and the first target grid cluster can include a grid cluster corresponding to the soil wall or a grid cluster corresponding to the gravel. Then, the first target grid cluster is further subjected to more detailed classification according to a preset classification condition, according to the height value of the first target grid cluster, whether there is a ground category grid around, whether it is on the lane or on one side of the lane, to respectively obtain a first target grid cluster of a first category obstacle (gravel) and a first target grid cluster of a second category obstacle (soil wall), and the second target grid cluster can be directly confirmed as the second category obstacle (soil wall), and the corresponding obstacles are respectively displayed in the reference semantic map.

[0061] The obstacle classification method of the application comprises the following steps: obtaining a grid result corresponding to a surrounding area of a current position of a target device based on data collected by at least one sensor on the target device, each grid in the grid result being provided with category information and a height value; determining candidate category grids from the grid result according to the category information and the height value; clustering the candidate category grids to obtain a plurality of candidate category grid clusters, wherein each candidate category grid cluster comprises at least one candidate category grid; and classifying the candidate category grid clusters based on a preset classification condition and a preset reference semantic map to determine a first category obstacle and a second category obstacle, wherein the preset classification condition is associated with position information and height values of the candidate category grid clusters, the preset reference semantic map comprises the surrounding area of the current position of the target device, can process data collected by the sensor, and classifies obstacles in the surrounding area, thereby improving the accuracy of obstacle classification and the safety of vehicle driving.

[0062] Corresponding to the foregoing application function implementation method embodiments, the application further provides an obstacle classification device, an electronic device, and corresponding embodiments.

[0063] Figure 2 FIG. 1 is a structural schematic diagram of an obstacle classification device according to an embodiment of the application.

[0064] Referring to Figure 2 An obstacle classification device 200 comprises: An acquisition module 210 is configured to obtain a grid result corresponding to a surrounding area of a current position of a target device based on data collected by at least one sensor on the target device, each grid in the grid result being provided with category information and a height value.

[0065] A determination module 220 is configured to determine candidate category grids from the grid result according to the category information and the height value.

[0066] A clustering module 230 is configured to cluster the candidate category grids to obtain a plurality of candidate category grid clusters, wherein each candidate category grid cluster comprises at least one candidate category grid. A classification module 240 is configured to classify the candidate category grid clusters based on a preset classification condition and a preset reference semantic map to obtain a first category obstacle and a second category obstacle, wherein the preset classification condition is associated with position information and height values of the candidate category grid clusters, and the preset reference semantic map comprises the surrounding area of the current position of the target device.

[0067] In a possible implementation, the acquisition module 210 is further configured to input data collected by the at least one sensor in real time into the multi-modal neural network model to obtain a grid result output by the multi-modal neural network model, where the data collected by the sensor in real time includes point cloud data, image data, and radar data of a surrounding area of a current position of the target device.

[0068] In a possible implementation, the determination module 220 is further configured to determine, from the grid result, a first category grid according to the category information of each grid; and display a third category obstacle in the reference semantic map based on the category information of the first category grid, where the third category obstacle includes a vehicle and sand.

[0069] In a possible implementation, the determination module 220 is further configured to determine, from the grid result, a second category grid according to the category information of each grid; perform a sliding average processing on an initial height value of the second category grid to obtain an average height value of the second category grid; determine, as a candidate category grid, a second category grid whose average height value is greater than a first preset height threshold; and determine, as a ground category grid, a second category grid whose average height value is not greater than the first preset height threshold.

[0070] In a possible implementation, the determination module 220 is further configured to include, in a formula for performing the sliding average processing on the second category grid:

[0071] wherein, is a target weight, the target weight is positively correlated with a number of observations of a current grid, is the average height value of the current grid, is an initial height value of the current grid.

[0072] In a possible implementation, the clustering module 230 is further configured to perform island filtering on the candidate category grid to obtain a plurality of candidate category grid clusters; generate a minimum bounding rectangle of each candidate category grid cluster, and obtain a diagonal line length of the minimum bounding rectangle; determine, as a first target grid cluster, a candidate category grid cluster whose diagonal line length is not greater than a preset length threshold; and determine, as a second target grid cluster, a candidate category grid cluster whose diagonal line length is greater than the preset length threshold.

[0073] In a possible implementation, the classification module 240 is further configured to: if the first target grid cluster is located on one side of a lane in the reference semantic map, and there is a ground category grid within a preset distance range of the first target grid cluster, and the average height value of the first target grid cluster is greater than a second preset height value, determine the first target grid cluster as a first category obstacle, and display the first category obstacle in the reference semantic map; if the first target grid cluster is located on the lane in the reference semantic map, and there is no ground category grid within the preset distance range of the first target grid cluster, and the average height value of the first target grid cluster is greater than a first preset height value, determine the first target grid cluster as a second category obstacle, and display the second category obstacle in the reference semantic map; and determine the second target grid cluster as the second category obstacle, and display the first category obstacle in the reference semantic map.

[0074] The obstacle classification device provided in the application comprises an acquisition module, a determination module, a clustering module, and a classification module. The acquisition module is configured to obtain a grid result corresponding to a surrounding area of a current position of a target device based on data collected by at least one sensor on the target device, and each grid in the grid result has category information and a height value. The determination module is configured to determine candidate category grids from the grid result according to the category information and the height value. The clustering module is configured to cluster the candidate category grids to obtain a plurality of candidate category grid clusters, wherein each candidate category grid cluster includes at least one candidate category grid. The classification module is configured to classify the candidate category grid clusters based on a preset classification condition and a preset reference semantic map to obtain a first category obstacle and a second category obstacle. The preset classification condition is associated with position information and height values of the candidate grid clusters. The preset reference semantic map includes the surrounding area of the current position of the target device, can process data obtained by the sensor, and can classify obstacles in the surrounding area, thereby improving the accuracy of obstacle classification and the safety of vehicle driving.

[0075] As to the device in the above-mentioned embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments of the method, and will not be described in detail here.

[0076] The application further provides an electronic device. Figure 3 FIG. 1 is a schematic diagram of a hardware structure of an embodiment of the electronic device. The electronic device comprises a memory 320 and at least one processor 310. The memory 320 is electrically connected to the at least one processor 310. The memory 320 stores instructions. The at least one processor 310 invokes the instructions in the memory 320, so that the electronic device performs the obstacle classification method according to any one of the preceding embodiments of the application.

[0077] In particular, the processor 310 described above can include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or can be configured as one or more integrated circuits to implement the embodiments of the present application.

[0078] The memory 320 can include mass storage for data or instructions. By way of example, and not limitation, the memory 320 can include a hard disk drive (HDD), a floppy disk drive, a flash memory, a compact disc (CD) or other optical disk, a tape drive, a universal serial bus (USB) drive or other suitable memory component, or a combination of two or more of these. Where appropriate, the memory 320 can include removable or non-removable (or fixed) media. Where appropriate, the memory 320 can be internal or external to the integrated gateway disaster recovery device. In certain embodiments, the memory 320 is nonvolatile memory. In certain embodiments, the memory 320 includes read-only memory (ROM). Where appropriate, this ROM can be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0079] In one example, the control device also includes a communication interface 330 and a bus 340. The processor 310, the memory 320, and the communication interface 330 are connected through the bus 340 and accomplish communication with each other.

[0080] The communication interface 330 is mainly used to realize the communication between the modules, devices, units and / or equipment in the embodiments of the present application.

[0081] The bus 340 includes hardware, software or both to couple components of the online data traffic billing device to each other and to control communication between the components. By way of example, and not limitation, the bus can include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or interconnect, or a combination of two or more of these. Where appropriate, the bus 340 can include one or more buses. Although the present application describes and illustrates a particular bus, this application contemplates any suitable bus or interconnect.

[0082] In addition, in combination with the obstacle classification method in the above-mentioned embodiments, the embodiments of the present application can provide a computer readable storage medium for implementation. The computer readable storage medium has instructions stored thereon, and the instructions are executed by a processor to implement any of the obstacle classification methods in the above-mentioned embodiments.

[0083] The present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of well-known methods are omitted herein. In the above-mentioned embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order of the steps, after understanding the spirit of the present application.

[0084] The functional blocks shown in the above structural block diagrams can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are program or code segments used to perform the required tasks. The program or code segments can be stored in a machine readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. The "machine readable medium" can include any medium capable of storing or transmitting information. Examples of the machine readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segments can be downloaded via a computer network such as the Internet, an intranet, etc.

[0085] It should also be noted that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above-mentioned steps, that is, the steps can be performed in the order mentioned in the embodiments, or in an order different from the embodiments, or several steps can be performed simultaneously.

[0086] Alternatively, the present application also provides a computer program product capable of implementing part or all of each step of the above-mentioned embodiments, and the computer program product includes a computer program / instruction, which, when executed by a processor, implements part or all of each step of the above-mentioned embodiments.

[0087] The above is only a specific implementation method of the present application. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited to this. Any technician familiar with this technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the scope of protection of this application.

Claims

1. An obstacle classification method, characterized in that: include: Based on data collected in real time by at least one sensor on the target device, a grid result corresponding to an area surrounding the current location of the target device is obtained, wherein each grid in the grid result has category information and a height value; Determining a candidate category grid from the grid results according to the category information and the height value; Clustering the candidate category grids to obtain a plurality of candidate category grid clusters, wherein the candidate category grid clusters include at least one candidate category grid; Based on preset classification conditions and a preset reference semantic map, the candidate category grid cluster is classified to determine first category obstacles and second category obstacles, the preset classification conditions are associated with the location information and height value of the candidate category grid cluster, and the preset reference semantic map includes the surrounding area of ​​the current location of the target device.

2. The method according to claim 1, characterized in that The obtaining of a grid result corresponding to an area surrounding a current location of the target device based on data collected in real time by at least one sensor on the target device includes: Inputting data collected in real time by at least one sensor into a multimodal neural network model to obtain the raster result output by the multimodal neural network model; wherein the data collected in real time by the sensor includes: point cloud data, image data and radar data of the surrounding area of ​​the current position of the target device.

3. The method according to claim 1, characterized in that Also includes: Determining a first category grid from the grid results according to the category information of each grid; Based on the category information of the first category grid, third category obstacles are displayed in the reference semantic map; wherein the third category obstacles include: vehicles and wind and sand.

4. The method according to claim 1, wherein Determining a candidate category grid from the grid results according to the category information and the height value includes: Determining a second category grid from the grid results according to the category information of each grid; Performing a sliding average process on the initial height values ​​of the second category grids to obtain an average height value of the second category grids; Determine the second category grid whose average height value is greater than the first preset height threshold as the candidate category grid; The second category grid whose average height value is not greater than the first preset height threshold is determined as a ground category grid.

5. The method according to claim 4, characterized in that The performing sliding average processing on the height values ​​of the second category grids to obtain the average height value of the second category grids includes: The formula for performing sliding average processing on the second category grid includes: in, is the target weight, which is positively correlated with the number of observations of the current grid. is the average height of the current grid, The initial height of the current grid.

6. The method according to claim 4, characterized in that The candidate category grids are clustered to obtain a plurality of candidate category grid clusters, including: Performing island filtering on the candidate category grids to obtain a number of candidate category grid clusters; Generate a minimum bounding rectangle for each candidate category grid cluster, and obtain the diagonal length of the minimum bounding rectangle; Determine the candidate category grid cluster whose diagonal length is not greater than a preset length threshold as the first target grid cluster; The candidate category grid cluster whose diagonal length is greater than a preset length threshold is determined as the second target grid cluster.

7. The method according to claim 6, characterized in that The step of classifying the candidate category grid clusters based on a preset classification condition and a preset reference semantic map to obtain first category obstacles and second category obstacles includes: If the first target grid cluster is located on one side of a lane in the reference semantic map, and the ground category grid exists within a preset distance range of the first target grid cluster, and the average height value of the first target grid cluster is greater than a second preset height value, then the first target grid cluster is determined to be a second category obstacle, and the second category obstacle is displayed in the reference semantic map; If the first target grid cluster is located on a lane in the reference semantic map, and no ground-type grid exists within a preset distance range of the first target grid cluster, and an average height value of the first target grid cluster is greater than a third preset height value, determining the first target grid cluster as a first-category obstacle and displaying the first-category obstacle in the reference semantic map; The second target grid cluster is determined as the second category of obstacles, and the second category of obstacles corresponding to the second target grid cluster are displayed in the reference semantic map.

8. The method according to claim 1, characterized in that include: The first category of obstacles is rubble, and the second category of obstacles is earth wall.

9. An obstacle classification device, characterized in that: include: An acquisition module is configured to obtain a grid result corresponding to an area surrounding a current location of a target device based on data collected in real time by at least one sensor on the target device, wherein each grid in the grid result has category information and a height value; A determination module, configured to determine a candidate category grid from the grid results according to the category information and the height value; A clustering module is used to cluster the candidate category grids to obtain a plurality of candidate category grid clusters, wherein the candidate category grid clusters include at least one candidate category grid, A classification module is used to classify the candidate category grid clusters based on preset classification conditions and a preset reference semantic map to obtain first category obstacles and second category obstacles, wherein the preset classification conditions are associated with the location information and height values ​​of the candidate category grid clusters, and the preset reference semantic map includes the surrounding area of ​​the current location of the target device.

10. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that An executable code is stored thereon, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the method according to any one of claims 1 to 8.

Images