Obstacle recognition method and device, electronic equipment, moving tool and storage medium
Through the feature fusion of image recognition data and point cloud data and the recognition of preset network models, combined with the prior knowledge of interference characteristics, the problem of false detection of lidar detection model is solved, the accuracy of obstacle detection is improved, and automatic The mistriggering of driving a car ensures the smooth and comfortable driving of the vehicle.
Patent Information
- Application Number
- CN202310580028.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-18
- Publication Date
- 2024-11-19
AI Technical Summary
The existing lidar detection model is prone to misdetecting trees and grass on the roadside as pedestrians or cyclists, and misdetecting car exhaust as vehicles, resulting in the automatic emergency braking function of autonomous vehicles being accidentally triggered, affecting the vehicle's driving stability and Comfort.
By obtaining image recognition data and foreground cloud data in driving scenarios, feature fusion is performed, and using preset network models for identification, combining the prior knowledge of interference characteristics, accurate obstacle information is output to reduce the misjudgment rate.
It improves the accuracy and robustness of obstacle detection, avoids the mistriggering of automatic emergency braking function caused by false detection, and ensures the smooth driving and comfort of the vehicle.
Smart Images

Figure CN118982808A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of autonomous driving technology, and in particular to an obstacle recognition method, device, electronic device, mobile tool and storage medium. Background Art
[0002] In order for autonomous vehicles to drive safely and stably on the road, they need to accurately perceive the surrounding environment so that they can make autonomous decisions on driving routes and safely operate the vehicle. At present, the method used by autonomous vehicles to identify obstacles is to use laser radar to obtain point cloud data, and then use the point cloud data obtained by the laser radar as the input of the laser detection model, and obtain the obstacle information corresponding to the point cloud data after the laser radar detection model is inferred.
[0003] However, in actual applications, existing lidar detection models are prone to misdetecting point cloud data of roadside trees and grass as pedestrians or cyclists, and are also prone to misdetecting point cloud data corresponding to car exhaust as vehicles. This can cause the automatic emergency braking (AEB) function of self-driving cars to be falsely triggered, affecting the smoothness and comfort of the vehicle's driving. Summary of the invention
[0004] In order to solve the above technical problems, the present application provides an obstacle recognition method, device, electronic device, mobile tool and storage medium.
[0005] In a first aspect, the present application provides an obstacle identification method, comprising:
[0006] Obtain image recognition data and foreground cloud data in driving scenarios;
[0007] Performing feature fusion on the foreground point cloud data and the image recognition data to obtain foreground point cloud fusion data;
[0008] The foreground point cloud fusion data is input into a preset network model, and obstacle information corresponding to the foreground point cloud fusion data is output, wherein the preset network model inserts prior knowledge corresponding to interference features in a driving scenario, and the interference features include non-human features, non-animal features, and non-mobile tool features.
[0009] In a second aspect, the present application provides an obstacle recognition device, comprising:
[0010] An acquisition module is used to acquire image recognition data and foreground cloud data in a driving scene;
[0011] A fusion module, used for performing feature fusion on the foreground point cloud data and the image recognition data to obtain foreground point cloud fusion data;
[0012] A detection module is used to input the foreground point cloud fusion data into a preset network model and output obstacle information corresponding to the foreground point cloud fusion data, wherein the preset network model inserts prior knowledge corresponding to interference features in the driving scene, and the interference features include non-human features, non-animal features, and non-mobile tool features.
[0013] In a third aspect, the present application provides a mobile tool, comprising the obstacle identification device provided in the second aspect above.
[0014] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:
[0015] Obtain image recognition data and foreground cloud data in driving scenarios;
[0016] Performing feature fusion on the foreground point cloud data and the image recognition data to obtain foreground point cloud fusion data;
[0017] The foreground point cloud fusion data is input into a preset network model, and obstacle information corresponding to the foreground point cloud fusion data is output, wherein the preset network model inserts prior knowledge corresponding to interference features in a driving scenario, and the interference features include non-human features, non-animal features, and non-mobile tool features.
[0018] Based on the above obstacle recognition method, the foreground point cloud data and image recognition data in the driving scene are feature fused to obtain foreground point cloud fusion data, which can overcome the detection limitations of a single sensor for obstacles, and use a preset network model combined with the corresponding prior knowledge of interference features to identify and process the foreground point cloud fusion data to output accurate obstacle information, which can reduce the misjudgment rate of misdetecting interference features as obstacles, and avoid the automatic emergency braking function of the self-driving car being mistakenly triggered due to false detection, thereby affecting the stability and comfort of the vehicle's driving. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0021] Figure 1 is a flow chart of an obstacle identification method in one embodiment;
[0022] Figure 2 is a flow chart of an obstacle identification method in one embodiment;
[0023] Figure 3 is a structural block diagram of an obstacle identification device in one embodiment;
[0024] Figure 4 FIG. 4 is a diagram showing the internal structure of a mobile tool in one embodiment. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0026] In one embodiment, Figure 1 FIG. 1 is a flow chart of an obstacle identification method in an embodiment, referring to FIG. Figure 1 , provides an obstacle recognition method. This embodiment mainly applies this method to an obstacle recognition device, and the obstacle recognition device can be mounted on a mobile tool. The mobile tool can be a vehicle with automatic driving or intelligent driving (including vehicles with manned functions (such as cars, buses, coaches, minibuses, etc.), vehicles with cargo functions (such as ordinary trucks, vans, trailers, closed trucks, tank trucks, flatbed trucks, container trucks, dump trucks, trucks with special structures), special vehicles (such as logistics distribution vehicles, automatic guided transport vehicles AGV, patrol cars, cranes, cranes, excavators, bulldozers, forklifts, rollers, loaders, off-road engineering vehicles, armored engineering vehicles, sewage treatment vehicles, sanitation vehicles, vacuum trucks, floor washing vehicles, sprinkler trucks, sweeping robots, food delivery robots, shopping guide robots, lawn mowers, golf carts, etc.), vehicles with entertainment functions (such as entertainment vehicles, playground automatic driving devices, balance cars, etc.), rescue vehicles (such as fire trucks, ambulances, power repair vehicles, engineering rescue vehicles, etc.)) and robots (such as sweeping robots, food delivery robots, etc.).
[0027] The obstacle recognition method specifically comprises the following steps:
[0028] Step S102, obtaining image recognition data and foreground point cloud data in a driving scene.
[0029] Specifically, the driving scene refers to any scene encountered by the mobile tool on the driving path, the image recognition data is used to indicate the pre-processed data of the captured image taken by the camera, and the foreground point cloud data is used to indicate the pre-processed data of the point cloud data collected by the lidar.
[0030] Step S104 , performing feature fusion on the foreground point cloud data and the image recognition data to obtain foreground point cloud fusion data.
[0031] Specifically, since lidar can provide high-resolution and long-distance three-dimensional point cloud data under severe weather conditions, and cameras can obtain the outline, texture, and color distribution of obstacles but cannot obtain depth information, the combination of lidar and cameras can make up for the limitations of a single device in identifying obstacles. That is, the foreground point cloud fusion data after feature fusion of foreground point cloud data and image recognition data can improve redundancy in detection accuracy, thereby improving the detection accuracy and robustness of mobile tools for obstacles.
[0032] Step S106, inputting the foreground point cloud fusion data into a preset network model, and outputting obstacle information corresponding to the foreground point cloud fusion data, wherein the preset network model inserts prior knowledge corresponding to interference features in the driving scene, and the interference features include non-human features, non-animal features, and non-mobile tool features.
[0033] Specifically, the preset network model is used to identify and process the foreground point cloud fusion data. Since the preset network model is combined with the prior knowledge corresponding to the interference features, the preset network model can accurately identify the interference features when identifying and processing the foreground point cloud fusion data. The accurately output obstacle information does not contain interference features. The interference features are custom features of non-mobile tools, non-human or non-animal objects such as trees, grass, and car exhaust, thereby avoiding misdetection of the point cloud data of roadside trees and grass as pedestrians or cyclists, or misdetection of the point cloud data corresponding to car exhaust as vehicles, resulting in the erroneous triggering of the automatic emergency braking function of the mobile tool, affecting the stability and comfort of the vehicle's driving.
[0034] The step of acquiring the image recognition data and the foreground point cloud data in the driving scene, i.e., step S102, specifically includes:
[0035] Acquire collected images and scanned point cloud data in driving scenes;
[0036] Performing feature recognition on the collected image to obtain the image recognition data, wherein the image recognition data includes pixel coordinates of each pixel point in the collected image and category values corresponding to each pixel coordinate;
[0037] The scanning point cloud data is subjected to ground fitting and filtering processing to obtain the foreground point cloud data.
[0038] Specifically, Figure 2 As shown, the captured image is a color image taken by a camera, and the scanned point cloud data is data collected by a laser radar. The camera and the laser radar are mounted on a mobile tool, and the models of the camera and the laser radar are not limited here. Feature recognition is performed on the captured image, that is, semantic segmentation based on deep learning is performed on the captured image to identify the feature categories in the captured image. The feature categories can specifically be background, road surface, curbstone, fence, wall, roadblock, guide sign, traffic sign, mobile tool, pedestrian, animal, etc. Different feature categories are represented by corresponding category values to obtain image recognition data. The image recognition data includes the category value corresponding to each pixel in the captured image, that is, the image recognition data is recorded as {I i}={(m,n,c) i T}, where i is used to indicate the number of the pixel in the acquired image, that is, the i-th pixel, (m, n) is used to indicate the two-dimensional pixel coordinates of the i-th pixel in the acquired image, c is used to indicate the category value corresponding to the i-th pixel, and the superscript T indicates matrix transposition.
[0039] The scanned point cloud data is three-dimensional data. In order to eliminate the interference of the ground on the subsequent obstacle detection, it is necessary to perform ground fitting filtering on the scanned point cloud data, that is, first perform ground fitting on the scanned point cloud data and then remove the fitted ground plane point cloud data to obtain the foreground point cloud data above the ground plane. The ground fitting method can be RANSAC method or grid-based method, etc. The foreground point cloud data is recorded as Among them, (x, y, z) j It refers to the 3D coordinates of the jth data point in the foreground point cloud data. Using the foreground point cloud data for subsequent obstacle detection can prevent the interference caused by the ground plane point cloud data and further improve the accuracy of the obstacle detection results.
[0040] In one embodiment, the step of performing feature fusion on the foreground point cloud data and the image recognition data to obtain foreground point cloud fusion data includes:
[0041] Projecting the foreground point cloud data into a two-dimensional image to obtain a projected image;
[0042] Determine the category value corresponding to each of the projection coordinates based on the mapping relationship between the projection coordinates of each pixel point in the projection image and the pixel coordinates of each pixel point in the image recognition data, and the category value corresponding to each pixel coordinate in the image recognition data;
[0043] The foreground point cloud fusion data is obtained based on the fusion of the foreground point cloud data and the category values corresponding to each of the projection coordinates, wherein the foreground point cloud fusion data includes fusion information corresponding to multiple projection coordinates, and the fusion information includes the projection coordinates, the category values corresponding to the projection coordinates, and the point cloud reflection intensity.
[0044] Specifically, the foreground point cloud data is projected into a two-dimensional image. The specific projection conversion formula is as follows:
[0045]
[0046] Among them, (X C ,Y C ,Z C ) is the coordinate of the foreground point cloud data converted to the camera coordinate system, (u, v) is the coordinate of the foreground point cloud data after conversion to the two-dimensional image, that is, the projection coordinate of the pixel point in the projected image, K is the calibrated camera intrinsic parameter matrix, T is the calibrated external parameter matrix between the laser radar and the camera, that is, the projected image is recorded as The projection coordinates of the pixel points in the projected image are mapped one by one with the pixel coordinates of the pixel points in the acquired image, thereby determining the category value corresponding to each projection coordinate based on the mapping relationship between the projection coordinates and the pixel coordinates and the category value corresponding to each pixel coordinate.
[0047] The foreground point cloud data is fused with the category value corresponding to the corresponding projection coordinates of each point in the foreground point cloud data to obtain the foreground point cloud fusion data. The foreground point cloud fusion data obtained by re-integrating the category value and the point cloud reflection intensity is: Where r represents the reflection intensity of the point cloud, so as to add the semantic segmentation information of the collected image to the foreground point cloud data. When the foreground point cloud fusion data is subsequently used for obstacle detection, the semantic segmentation information corresponding to the category value can improve the accuracy of the obstacle detection result.
[0048] In one embodiment, the determining the category value corresponding to each of the projection coordinates based on the mapping relationship between the projection coordinates of each pixel point in the projection image and the pixel coordinates of each pixel point in the image recognition data, and the category value corresponding to each pixel coordinate in the image recognition data, includes:
[0049] Using the category value corresponding to the pixel coordinates of each pixel point in the projection image as the category value corresponding to the projection coordinates that have a mapping relationship with the pixel coordinates;
[0050] When there are null-valued projection coordinates in the projection image, a preset value is used as a category value corresponding to the null-valued projection coordinate, wherein the null-valued projection coordinate is a projection coordinate without a corresponding category value.
[0051] Specifically, the projection coordinates that have a mapping relationship with the pixel coordinates corresponding to the category value correspond to the same category value, and the category value is used to indicate the feature. There may be a situation where some pixel coordinates have no corresponding category value, which results in the projection coordinates that have a mapping relationship with the pixel coordinates also having no corresponding category value. The projection coordinates without corresponding category values are taken as null-value projection coordinates. In order to make all projection coordinates correspond to corresponding category values for subsequent obstacle detection, a preset value is taken as the category value corresponding to the null-value projection coordinate. The preset value is a custom value that is different from the category value corresponding to any of the above-mentioned feature categories. In this embodiment, the preset value is set to 255, so as to determine the category value corresponding to each projection coordinate in the projection image, so as to avoid some projection coordinates having corresponding category values and some projection coordinates not having corresponding category values, which increases the difficulty of data processing for subsequent obstacle detection.
[0052] In one embodiment, the step of inputting the foreground point cloud fusion data into a preset network model and outputting obstacle information corresponding to the foreground point cloud fusion data includes:
[0053] Inputting the fusion information corresponding to each projection coordinate in the foreground point cloud fusion data into a preset model loss function in a preset network model for iterative calculation as a loss function value, wherein the preset interference feature data is data in the preset model loss function for indicating the prior knowledge corresponding to the interference feature, wherein the preset model loss function combines the prior knowledge corresponding to the interference feature in the driving scene;
[0054] When the loss function value meets the preset numerical requirement, the obstacle information corresponding to the foreground point cloud fusion data is output.
[0055] Specifically, the fusion information corresponding to each projection coordinate in the foreground point cloud fusion data includes the projection coordinate, the category value corresponding to the projection coordinate, and the point cloud reflection intensity. The foreground point cloud fusion data is input into the preset network model. The preset model loss function in the preset network model iteratively calculates the fusion information corresponding to each projection coordinate as the loss function value, wherein the preset interference feature data is the data used to indicate the prior knowledge corresponding to the interference feature in the preset model loss function, and the preset model loss function combines the prior knowledge corresponding to the interference feature. Therefore, the preset loss function can be used to accurately distinguish the interference feature and the obstacle feature. When the loss function value meets the preset numerical requirements, the obstacle information corresponding to the foreground point cloud fusion data is output, and the obstacle information includes the category, detection box, location information, etc. of the obstacle.
[0056] In one embodiment, the step of inputting the fusion information corresponding to each projection coordinate in the foreground point cloud fusion data into a preset model loss function in a preset network model for iterative calculation as the loss function value includes:
[0057] Determine the category prediction data, the detection box prediction data and the parameter prediction data corresponding to the foreground point cloud fusion data based on the preset model loss function in the preset network model;
[0058] The result of weighted summation of the parameter values of the preset interference feature data and the category prediction data, detection box prediction data, parameter prediction data, and parameter values of the preset interference feature data corresponding to the foreground point cloud fusion data is used as the loss function value, wherein the preset interference feature data is data in the preset model loss function used to indicate the prior knowledge corresponding to the interference feature.
[0059] Specifically, the calculation formula for category prediction data is as follows:
[0060]
[0061] Among them, N refers to the number of training samples during the training of the preset network model, and refers to the total number of data points in the foreground cloud fusion data during the use of the preset network model, which also refers to the total number of projection coordinates. C is the total number of feature categories, and y j,c represents the cth category label of the jth sample (data point / projection coordinate), It indicates the predicted probability that the jth sample is the cth feature category. Therefore, based on the category prediction data, the probability that each projection coordinate corresponding data point is a different feature category can be predicted.
[0062] The calculation formula for the detection box prediction data is as follows:
[0063]
[0064] in, Represents the intersection-over-union ratio between the true detection box and the predicted detection box of the jth sample.
[0065] The calculation formula for parameter prediction data is as follows:
[0066]
[0067] Among them, y j,s represents the true value of the sth parameter of the jth sample, Indicates the predicted value of the sth parameter of the jth sample, smooth L1It is the Smooth L1 function, where (x, y, z) represents the three-dimensional coordinates of the center point of the detection frame, r represents the average reflection intensity of all point clouds in the detection frame, c represents the category value corresponding to the obstacle category, (l, h, w) represents the length, width, and height of the detection frame, and θ represents the orientation angle of the detection frame.
[0068] The preset interference characteristic data is:
[0069]
[0070] Among them, Z is a 1*2 vector, j=0 represents interference features such as trees and grass, j=1 represents obstacles, and the preset interference feature data is the data customized and inserted by the model trainer for the learning requirements of the model. The preset interference feature data can be used to distinguish interference features and obstacles, and the accuracy of the model can be further improved by adding prior knowledge to the model.
[0071] The above-mentioned category prediction data, detection box prediction data, parameter prediction data and preset interference feature data are weighted and summed to obtain the loss function value of the preset loss function. The calculation formula is as follows:
[0072] L=α1L ce +α2L iou +α3L l1 +α4L plant
[0073] Among them, α1~α4 are weighted coefficients of category prediction data, detection box prediction data, parameter prediction data, and preset interference feature data, respectively. They can be customized according to the actual application scenario, and it is necessary to satisfy α1+α2+α3+α4=1. In this embodiment, α1=0.2, α2=0.3, α3=0.2, and α4=0.3.
[0074] In one embodiment, when the loss function value meets the preset numerical requirement, outputting the obstacle information corresponding to the foreground point cloud fusion data includes:
[0075] When the loss function value meets the preset numerical requirements, the obstacle information corresponding to the foreground point cloud fusion data is determined based on the category prediction data, the detection box prediction data, the parameter prediction data and the preset interference feature data corresponding to the loss function value.
[0076] Specifically, the preset numerical value requirement is used to indicate that the loss function value must be less than the loss threshold, or the loss function value is the minimum loss value. The final obstacle information is output based on the category prediction data, the detection frame prediction data, the parameter prediction data and the preset interference feature data, that is, the category of the obstacle can be determined based on the category prediction data, the detection frame containing the obstacle can be determined based on the detection frame prediction data, and the accuracy of each parameter can be determined based on the parameter prediction data. The parameters can specifically be coordinates, point cloud reflection intensity, category value, detection frame size, and detection frame orientation angle. Based on the preset interference feature data, obstacles and interference features can be clearly distinguished to prevent interference features from being detected as obstacles. In this way, obstacle information is determined by combining the preset loss function with the corresponding prior knowledge of the interference features, which can reduce the misjudgment rate of determining interference features as obstacles and avoid affecting the stability and comfort of the mobile tool during driving.
[0077] Figure 1 and Figure 2 FIG. 1 is a flow chart of an obstacle identification method in one embodiment. It should be understood that although Figure 1 and Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 and Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0078] In one embodiment, Figure 3 As shown, an obstacle recognition device is provided, comprising:
[0079] An acquisition module 310 is used to acquire image recognition data and foreground point cloud data in a driving scene;
[0080] A fusion module 320 is used to perform feature fusion on the foreground point cloud data and the image recognition data to obtain foreground point cloud fusion data;
[0081] The detection module 330 is used to input the foreground point cloud fusion data into a preset network model and output obstacle information corresponding to the foreground point cloud fusion data, wherein the preset network model inserts prior knowledge corresponding to interference features in the driving scene, and the interference features include non-human features, non-animal features, and non-mobile tool features.
[0082] In one embodiment, the fusion module 320 is specifically used for:
[0083] Projecting the foreground point cloud data into a two-dimensional image to obtain a projected image;
[0084] Determine the category value corresponding to each of the projection coordinates based on the mapping relationship between the projection coordinates of each pixel point in the projection image and the pixel coordinates of each pixel point in the image recognition data, and the category value corresponding to each pixel coordinate in the image recognition data;
[0085] The foreground point cloud fusion data is obtained based on the fusion of the foreground point cloud data and the category values corresponding to each of the projection coordinates, wherein the foreground point cloud fusion data includes fusion information corresponding to multiple projection coordinates, and the fusion information includes the projection coordinates, the category values corresponding to the projection coordinates, and the point cloud reflection intensity.
[0086] In one embodiment, the fusion module 320 is further configured to:
[0087] Using the category value corresponding to the pixel coordinates of each pixel point in the projection image as the category value corresponding to the projection coordinates that have a mapping relationship with the pixel coordinates;
[0088] When there are null-valued projection coordinates in the projection image, a preset value is used as a category value corresponding to the null-valued projection coordinate, wherein the null-valued projection coordinate is a projection coordinate without a corresponding category value.
[0089] In one embodiment, the detection module 330 is further configured to:
[0090] Inputting the fusion information corresponding to each projection coordinate in the foreground point cloud fusion data into a preset model loss function in a preset network model for iterative calculation as a loss function value, wherein the preset interference feature data is data in the preset model loss function for indicating the prior knowledge corresponding to the interference feature, wherein the preset model loss function combines the prior knowledge corresponding to the interference feature in the driving scene;
[0091] When the loss function value meets the preset numerical requirement, the obstacle information corresponding to the foreground point cloud fusion data is output.
[0092] In one embodiment, the detection module 330 is further configured to:
[0093] Determine the category prediction data, the detection box prediction data and the parameter prediction data corresponding to the foreground point cloud fusion data based on the preset model loss function in the preset network model;
[0094] The result of weighted summation of the parameter values of the preset interference feature data and the category prediction data, detection box prediction data, parameter prediction data, and parameter values of the preset interference feature data corresponding to the foreground point cloud fusion data is used as the loss function value, wherein the preset interference feature data is data in the preset model loss function used to indicate the prior knowledge corresponding to the interference feature.
[0095] In one embodiment, the detection module 330 is further configured to:
[0096] When the loss function value meets the preset numerical requirements, the obstacle information corresponding to the foreground point cloud fusion data is determined based on the category prediction data, the detection box prediction data, the parameter prediction data and the preset interference feature data corresponding to the loss function value.
[0097] In one embodiment, the acquisition module 310 is further used for:
[0098] Acquire collected images and scanned point cloud data in driving scenes;
[0099] Performing feature recognition on the collected image to obtain the image recognition data, wherein the image recognition data includes pixel coordinates of each pixel point in the collected image and category values corresponding to each pixel coordinate;
[0100] The scanning point cloud data is subjected to ground fitting and filtering processing to obtain the foreground point cloud data.
[0101] Figure 4 FIG. 2 shows an internal structure diagram of a mobile tool in one embodiment. Figure 4 As shown, the mobile tool includes a processor, a memory, a network interface, an input device and a display screen connected via a system bus. Among them, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the mobile tool stores an operating system and may also store a computer program, which, when executed by the processor, enables the processor to implement an obstacle recognition method. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to execute an obstacle recognition method. The display screen of the mobile tool may be a liquid crystal display screen or an electronic ink display screen, and the input device of the mobile tool may be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the mobile tool, or an external keyboard, touchpad or mouse, etc.
[0102] Those skilled in the art will understand that Figure 4The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the mobile tool to which the scheme of the present application is applied. The specific mobile tool may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0103] In one embodiment, the obstacle identification device provided by the present application can be implemented in the form of a computer program. The computer program can be used in Figure 4 The memory of the mobile tool may store various program modules constituting the obstacle recognition device, for example, Figure 3 The acquisition module 310, the fusion module 320 and the detection module 330 are shown. The computer program composed of various program modules enables the processor to execute the steps of the obstacle recognition method of each embodiment of the present application described in this specification.
[0104] Figure 4 The moving tools shown can be Figure 3 The acquisition module 310 in the obstacle recognition device shown acquires the image recognition data and foreground point cloud data in the driving scene. The mobile tool can perform feature fusion of the foreground point cloud data and the image recognition data through the fusion module 320 to obtain foreground point cloud fusion data. The mobile tool can input the foreground point cloud fusion data into a preset network model through the detection module 330, and output the obstacle information corresponding to the foreground point cloud fusion data, wherein the preset network model inserts the prior knowledge corresponding to the interference features in the driving scene, and the interference features include non-human features, non-animal features, and non-mobile tool features.
[0105] In one embodiment, an electronic device is provided. The electronic device can be loaded on a mobile tool, and includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method described in any one of the above embodiments is implemented.
[0106] In a specific implementation, the mobile tool may be equipped with various sensors for sensing environmental information and the mobile tool's own information (including but not limited to lidar, camera, millimeter-wave radar, ultrasonic radar, inertial measurement unit, wheel speed meter, etc.), and the electronic device may be an autonomous driving domain controller mounted on the mobile tool. When the mobile tool is in working state, the autonomous driving domain controller communicates with the various sensors mounted on the mobile tool, and plans a path for the mobile tool based on the environmental information collected by these sensors and the mobile tool's own information, so as to achieve the purpose of controlling the mobile tool to drive automatically.
[0107] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in any one of the above embodiments is implemented.
[0108] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0109] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0110] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.
Claims
1. An obstacle recognition method, characterized in that: The method comprises: Obtain image recognition data and foreground cloud data in driving scenarios; Performing feature fusion on the foreground point cloud data and the image recognition data to obtain foreground point cloud fusion data; The foreground point cloud fusion data is input into a preset network model, and obstacle information corresponding to the foreground point cloud fusion data is output, wherein the preset network model inserts prior knowledge corresponding to interference features in a driving scenario, and the interference features include non-human features, non-animal features, and non-mobile tool features.
2. The method according to claim 1, characterized in that The step of fusing the foreground point cloud data and the image recognition data to obtain foreground point cloud fusion data includes: Projecting the foreground point cloud data into a two-dimensional image to obtain a projected image; Determine the category value corresponding to each of the projection coordinates based on the mapping relationship between the projection coordinates of each pixel point in the projection image and the pixel coordinates of each pixel point in the image recognition data, and the category value corresponding to each pixel coordinate in the image recognition data; The foreground point cloud fusion data is obtained based on the fusion of the foreground point cloud data and the category values corresponding to each of the projection coordinates, wherein the foreground point cloud fusion data includes fusion information corresponding to multiple projection coordinates, and the fusion information includes the projection coordinates, the category values corresponding to the projection coordinates, and the point cloud reflection intensity.
3. The method according to claim 2, characterized in that The determining, based on a mapping relationship between the projection coordinates of each pixel point in the projection image and the pixel coordinates of each pixel point in the image recognition data, and a category value corresponding to each pixel coordinate in the image recognition data, the category value corresponding to each projection coordinate includes: Using the category value corresponding to the pixel coordinates of each pixel point in the projection image as the category value corresponding to the projection coordinates that have a mapping relationship with the pixel coordinates; When there are null-valued projection coordinates in the projection image, a preset value is used as a category value corresponding to the null-valued projection coordinate, wherein the null-valued projection coordinate is a projection coordinate without a corresponding category value.
4. The method according to claim 2, characterized in that: The step of inputting the foreground point cloud fusion data into a preset network model and outputting obstacle information corresponding to the foreground point cloud fusion data includes: Inputting the fusion information corresponding to each projection coordinate in the foreground point cloud fusion data into a preset model loss function in a preset network model for iterative calculation as a loss function value, wherein the preset interference feature data is data in the preset model loss function for indicating the prior knowledge corresponding to the interference feature, wherein the preset model loss function combines the prior knowledge corresponding to the interference feature in the driving scene; When the loss function value meets the preset numerical requirement, the obstacle information corresponding to the foreground point cloud fusion data is output.
5. The method according to claim 4, characterized in that The step of inputting the fusion information corresponding to each projection coordinate in the foreground point cloud fusion data into a preset model loss function in a preset network model for iterative calculation as a loss function value includes: Determine the category prediction data, the detection box prediction data and the parameter prediction data corresponding to the foreground point cloud fusion data based on the preset model loss function in the preset network model; The result of weighted summation of the parameter values of the preset interference feature data and the category prediction data, detection box prediction data, parameter prediction data, and parameter values of the preset interference feature data corresponding to the foreground point cloud fusion data is used as the loss function value, wherein the preset interference feature data is data in the preset model loss function used to indicate the prior knowledge corresponding to the interference feature.
6. The method according to claim 5, characterized in that When the loss function value meets the preset numerical requirement, the obstacle information corresponding to the foreground point cloud fusion data is output, including: When the loss function value meets the preset numerical requirements, the obstacle information corresponding to the foreground point cloud fusion data is determined based on the category prediction data, the detection box prediction data, the parameter prediction data and the preset interference feature data corresponding to the loss function value.
7. The method according to claim 1, characterized in that The obtaining of image recognition data and foreground point cloud data in the driving scene includes: Acquire collected images and scanned point cloud data in driving scenes; Performing feature recognition on the collected image to obtain the image recognition data, wherein the image recognition data includes pixel coordinates of each pixel point in the collected image and category values corresponding to each pixel coordinate; The scanning point cloud data is subjected to ground fitting and filtering processing to obtain the foreground point cloud data.
8. An obstacle recognition device, characterized in that: The device comprises: An acquisition module is used to acquire image recognition data and foreground cloud data in a driving scene; A fusion module, used for performing feature fusion on the foreground point cloud data and the image recognition data to obtain foreground point cloud fusion data; A detection module is used to input the foreground point cloud fusion data into a preset network model and output obstacle information corresponding to the foreground point cloud fusion data, wherein the preset network model inserts prior knowledge corresponding to interference features in the driving scene, and the interference features include non-human features, non-animal features, and non-mobile tool features.
9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of any one of claims 1-7.
10. A mobile tool, characterized in that: An electronic device comprising the electronic device described in claim 9.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Human body action recognition method based on a TP-STG framework
CN109492581A
Obstacle detection method and device, computer equipment and storage medium
CN111191600A
Automatic driving environment sensing method and system
CN112101092A
Object identification method and device
CN114550116A
Mining area environment sensing method based on 4D millimeter wave radar
CN115236674A
