Automatic driving obstacle detection and identification method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of automatic driving, and provides an automatic driving obstacle detection and identification method and device, electronic equipment and a storage medium; the method comprises the following steps: detecting obstacles in front of a vehicle, and performing first deceleration control according to a detection result; collecting data of the obstacles in front of the vehicle after the first deceleration, fusing the data collection result with the detection result to model, and performing second deceleration control according to a fusion modeling result; detecting and analyzing the obstacles in front of the vehicle after the second deceleration, and keeping or changing a vehicle driving state according to a detection and analysis result. The application detects and identifies the obstacles in front of the vehicle through hierarchical detection and identification, so that the vehicle can detect and identify the obstacles in front of the vehicle even in a poor observation condition, and the vehicle speed or the vehicle driving state can be adjusted in time, which not only can reduce wear caused by emergency braking of the vehicle, but also can guarantee effective obstacle avoidance of the vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to an automatic driving obstacle detection and identification method and device, an electronic device and a storage medium. BACKGROUND

[0002] Driving a mining truck is relatively dangerous. In the open pit mine environment, using reliable automatic driving technology to replace manual driving can not only effectively reduce the harm of dust and high temperature to the health of the driver, but also greatly improve the operation efficiency of the mine truck and reduce the labor cost. The mine truck is large in size, heavy in load and far in braking distance. Taking the Xiangtan Heavy Industry MCC600D heavy mine dump truck as an example, the self-weight is 180 tons, and the total weight reaches 600 tons under full load. The braking distance of the truck is 100m when it travels at a speed of 50Km / h under full load. In order to ensure the collision safety distance of the dump truck, the experienced driver needs to observe far enough to make a detour or brake in advance when driving manually. The automatic driving requires that the obstacle detection capability of the sensor is at least 100 meters. Detecting obstacles is an essential part of automatic driving, and its accuracy directly affects the level of automatic driving and even the usability of the unmanned system.

[0003] In the prior art, the obstacle detection and identification algorithm applied to automatic driving of a vehicle is based on laser and vision, and the vehicle is controlled after detecting the obstacle. However, the above obstacle detection method for automatic driving can only be applied to highway vehicles and cannot be applied to mine trucks. In the case of poor observation conditions in the mine area, such as dust interference, the detection distance of the conventional laser radar sensor is limited. When the laser radar sensor detects an obstacle, the distance between the mine truck and the obstacle is already very small, which will cause the mine truck to frequently brake hard, resulting in wear and tear and making it difficult to avoid obstacles.

[0004] At present, there is no effective technical solution to the above problems. SUMMARY

[0005] The present application aims to provide an automatic driving obstacle detection and identification method, device, electronic device and storage medium, so that the vehicle can still detect and identify the obstacles in front of the vehicle under poor observation conditions, and adjust the vehicle speed or vehicle driving state in time, which can not only reduce the wear and tear caused by the vehicle braking hard, but also ensure the effective obstacle avoidance of the vehicle.

[0006] In a first aspect, the present application provides an automatic driving obstacle detection and identification method for detecting and identifying obstacles in front of a vehicle, comprising the following steps:

[0007] detecting obstacles in front of the vehicle and performing first speed reduction control according to the detection result;

[0008] After the first deceleration, data of the obstacle in front of the vehicle is collected, the data collection result is fused with the detection result to model, and second deceleration control is performed according to the modeling result;

[0009] After the second deceleration, the obstacle in front of the vehicle is detected and analyzed, and the vehicle driving state is maintained or changed according to the detection and analysis result.

[0010] The automatic driving obstacle detection and identification method provided in the application detects and identifies the obstacle in front of the vehicle through hierarchical levels, so that the vehicle can still detect and identify the obstacle in front of the vehicle in the case of poor observation conditions, and timely adjust the vehicle speed or vehicle driving state, which not only reduces the wear caused by sudden braking, but also ensures effective obstacle avoidance of the vehicle.

[0011] Optionally, in the automatic driving obstacle detection and identification method provided in the embodiment of the application, the detection and analysis of the obstacle in front of the vehicle after the second deceleration comprises the following steps:

[0012] An enhanced image of the obstacle in front of the vehicle is obtained;

[0013] Multi-level deep features of the enhanced image are extracted;

[0014] The multi-level deep features are detected and analyzed.

[0015] The application detects and analyzes the obstacle in front of the vehicle to determine the material of the obstacle, and timely adjusts the vehicle state according to the material of the obstacle to avoid the influence of the obstacle on the vehicle driving.

[0016] Optionally, in the automatic driving obstacle detection and identification method provided in the application, after the multi-level deep features of the enhanced image are extracted and before the multi-level deep features are detected and analyzed, the following steps are further included:

[0017] The multi-level deep features are fused to obtain semantic features.

[0018] The application fuses the multi-level deep features, so that the size of the feature map is reduced and the information contained is more abundant, thereby improving the accuracy of detection and analysis.

[0019] Optionally, in the automatic driving obstacle detection and identification method provided in the application, the second deceleration control according to the modeling result comprises the following steps:

[0020] According to whether the modeling result is within a preset value range, it is judged whether to perform second deceleration control.

[0021] Optionally, in the autonomous driving obstacle detection and recognition method described in this application, the step of fusing and modeling the data acquisition results with the detection results includes the following steps:

[0022] Obtain the geographic reference information of the vehicle and the relative information of the vehicle's working environment with respect to the coordinate system of the laser scanner;

[0023] Coordinate transformation is performed on the geographic reference information and the relative information.

[0024] Optionally, in the autonomous driving obstacle detection and recognition method described in this application, the coordinate transformation of the geographic reference information and the relative information is calculated using the following formula:

[0025] ;

[0026] in, Let P be the coordinates of the laser scanning point P in the geocentric rectangular coordinate system; The coordinates of the IMU / GNSS center in the geocentric rectangular coordinate system are obtained from the position output measured by the IMU / GNSS system. The rotation matrix from the IMU / GNSS coordinate system to the geocentric rectangular coordinate system is formed by the attitude measured by the IMU / GNSS system; The component of the offset from the laser scanner center to the IMU / GNSS center in the IMU / GNSS coordinate system is expressed, and the initial value is obtained through manual measurement. The rotation matrix from the laser scanner coordinate system to the IMU / GNSS coordinate system is determined by the specific installation axis; These are the coordinates of the laser scanner's scan point in the laser scanner's coordinate system, output by the laser scanner.

[0027] Optionally, the autonomous driving obstacle detection and recognition method described in this application further includes the following steps before detecting and analyzing obstacles in front of the vehicle after the second deceleration:

[0028] Adjust the position of the device used to detect and analyze obstacles in front of the vehicle.

[0029] Secondly, this application also provides an autonomous driving obstacle detection and recognition device for detecting and recognizing obstacles in front of a vehicle, the device comprising:

[0030] The detection module is used to detect obstacles in front of the vehicle and perform the first deceleration control based on the detection results;

[0031] The data acquisition module is configured to acquire data of the obstacle in front of the vehicle after the first deceleration, fuse the data acquisition result with the detection result to build a model, and control the vehicle to decelerate for the second time according to the fusion modeling result.

[0032] The detection analysis module is configured to detect and analyze the obstacle in front of the vehicle after the second deceleration, and keep or change the driving state of the vehicle according to the detection analysis result.

[0033] The automatic driving obstacle detection and recognition device provided by the application can detect and recognize the obstacle in front of the vehicle through hierarchical detection and recognition, which can improve the ability of the vehicle to detect and recognize the obstacle in a poor observation condition, and can also adjust the speed of the vehicle to reduce the wear caused by sudden braking.

[0034] In a third aspect, the application provides an electronic device, comprising a processor and a memory, wherein the memory stores computer readable instructions, and when the computer readable instructions are executed by the processor, the steps in the method provided in the first aspect are executed.

[0035] In a fourth aspect, the application provides a storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps in the method provided in the first aspect are executed.

[0036] As can be seen from the above, the automatic driving obstacle detection and recognition method, device, electronic device and storage medium provided by the application can detect the obstacle in front of the vehicle according to the detection result, control the vehicle to decelerate for the first time, acquire data of the obstacle in front of the vehicle after the first deceleration, fuse the data acquisition result with the detection result to build a model, control the vehicle to decelerate for the second time according to the fusion modeling result, detect and analyze the obstacle in front of the vehicle after the second deceleration, and keep or change the driving state of the vehicle according to the detection analysis result. Through hierarchical detection and recognition of the obstacle in front of the vehicle, the vehicle can still detect and recognize the obstacle in front of the vehicle in a poor observation condition, and the speed or driving state of the vehicle can be adjusted in time, which not only reduces the wear caused by sudden braking, but also ensures effective obstacle avoidance of the vehicle.

[0037] Other features and advantages of the application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the application. The objects and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The flowchart of the automatic driving obstacle detection and recognition method provided by the embodiment of the application.

[0039] Figure 2 A fusion modeling schematic diagram of the automatic driving obstacle detection and identification method provided by the embodiment of the present application.

[0040] Figure 3 An image schematic diagram before three-dimensional modeling of the road surface in front of the vehicle provided by the embodiment of the present application.

[0041] Figure 4 An image schematic diagram after three-dimensional modeling of the road surface in front of the vehicle provided by the embodiment of the present application.

[0042] Figure 5 A structural schematic diagram of the automatic driving obstacle detection and identification device provided by the embodiment of the present application.

[0043] Figure 6 A structural schematic diagram of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0045] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms “first”, “second”, etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0046] In the case of poor observation conditions in the mining area, the detection distance of the conventional laser radar sensor is limited, the obstacle detection distance is greatly reduced, and the short distance detection will cause the mine truck to frequently and suddenly stop, which not only causes wear and tear, but also easily causes the situation of being unable to avoid obstacles. Based on this, the present application provides an automatic driving obstacle detection and identification method, device, electronic equipment and storage medium.

[0047] Please refer to Figure 1 , Figure 1is a flowchart of an automatic driving obstacle detection and identification method in some embodiments of the present application. The automatic driving obstacle detection and identification method of the present application is used to detect and identify obstacles in front of the vehicle, and control the vehicle, including the following steps:

[0048] S10, detecting obstacles in front of the vehicle, and performing first deceleration control according to the detection result;

[0049] S20, collecting data of the obstacles in front of the vehicle after the first deceleration, fusing the data collection result with the detection result to model, and performing second deceleration control according to the modeling result;

[0050] S30, detecting and analyzing the obstacles in front of the vehicle after the second deceleration, and maintaining or changing the driving state of the vehicle according to the detection and analysis result. Maintaining the driving state of the vehicle means that the vehicle continues to drive forward at the current speed, and changing the driving state of the vehicle means that the vehicle drives around or stops.

[0051] In this step S10, the millimeter wave radar can be used to detect obstacles in front of the vehicle. The millimeter wave has strong ability to penetrate fog, smoke and dust, and has the characteristics of all-weather (except heavy rain) and all-day. It can perceive whether there is an obstacle in the far distance range (300 meters in front of the vehicle) in front of the vehicle. It should be noted that the millimeter wave radar detects obstacles in front of the vehicle, including detecting static obstacles and detecting and tracking dynamic obstacles. By emitting millimeter waves to the front of the vehicle, it is judged whether there is an obstacle in front of the vehicle according to whether the millimeter wave radar receives a return wave (the return wave is millimeter wave point cloud data), so as to timely control the vehicle to decelerate for the first time. If there is an obstacle in front of the vehicle, the first deceleration control is to control the vehicle to decelerate; if there is no obstacle in front of the vehicle, the first deceleration control is to control the vehicle to maintain the current driving state and continue to drive without deceleration.

[0052] In this step S20, the laser radar can be used to collect data of the obstacles in front of the vehicle after the first deceleration. By emitting a detection signal to the obstacles in front of the vehicle, receiving a signal reflected from the obstacles, the reflected signal being laser radar point cloud data, and fusing the laser radar point cloud data with the millimeter wave point cloud data to model, the fusion modeling result, i.e. the parameters of the obstacles, is obtained. The parameters of the obstacles include distance, direction, height, speed, attitude and shape, etc. According to the comparison result of the parameters of the obstacles and the preset value range, the vehicle is controlled to decelerate for the second time. If the difference between the parameters of the obstacles and the preset value range is within the preset allowable range, the second deceleration control is to control the vehicle to maintain the current driving state and continue to drive without deceleration; if the difference between the parameters of the obstacles and the preset value range is not within the preset allowable range, the second deceleration control is to control the vehicle to decelerate.

[0053] In this step S30, the detection and analysis of the obstacle in front of the vehicle after the second deceleration can use a multispectral camera. The multispectral camera can simultaneously obtain the spatial and spectral information of the object, comprehensively detect and identify the characteristics of the object, and especially can finely identify the obstacle category and the road surface condition. Even if the overall road surface appears to be large-area water accumulation or icing, the multispectral camera can still finely segment and identify the overall condition of the road surface by using the difference in the reflectivity of light of different wavelengths under various road surface conditions.

[0054] In some embodiments, the step S20 includes the following sub-steps:

[0055] Obtaining the geographical reference information of the vehicle and the relative information of the working environment of the vehicle relative to the coordinate system of the laser scanner;

[0056] Converting the coordinates of the geographical reference information and the relative information.

[0057] The geographical reference information of the vehicle can be directly obtained by the IMU / GNSS system, and the IMU / GNSS system obtains point cloud data; the relative information of the working environment of the vehicle relative to the coordinate system of the laser scanner can be collected by the laser radar, and the laser radar collects point cloud data; the point cloud data of each frame of the geographical reference information and the relative information is converted from the IMU / GNSS coordinate system and the laser scanner coordinate system to the geocentric rectangular coordinate system by an interpolation algorithm, so as to realize the coordinate conversion of the geographical reference information and the relative information.

[0058] Please refer to Figure 2 , Figure 2 The fusion modeling schematic diagram of the automatic driving obstacle detection and identification method provided by the embodiments of the present application. Specifically, the coordinate conversion is calculated by the following formula: ;

[0059] Among them, is the coordinate of the laser scanning point P in the geocentric rectangular coordinate system (referred to as E system); is the coordinate of the IMU / GNSS center in the geocentric rectangular coordinate system, that is, the geographical reference information of the vehicle, which is obtained by the position output of the IMU / GNSS system; is the rotation matrix from the IMU / GNSS coordinate system (referred to as I system) to the geocentric rectangular coordinate system, which is formed by the attitude measured by the IMU / GNSS system; is the component expression of the bias of the scanning center of the laser scanner to the IMU / GNSS center in the IMU / GNSS coordinate system, and the initial value can be obtained by manual measurement; is the rotation matrix from the laser scanner coordinate system to the IMU / GNSS coordinate system, which is determined by the specific installation axis, that is, The specific installation position of the laser scanner determines; The coordinates of the scanning points of the laser scanner in the laser scanner coordinate system (referred to as S system) are the relative information of the vehicle working environment relative to the laser scanner coordinate system, which is output by the laser scanner.

[0060] Preferably, after the coordinate conversion, the point cloud can be processed by the visual sensor for gray shading, and a high-precision three-dimensional point cloud information with color information, road information and surrounding environment information can be constructed. Through the three-dimensional point cloud information, the parameters of the obstacles can be clearly obtained.

[0061] In some embodiments, step S20 further includes the following steps: detecting the road surface in front of the vehicle. The detection of the road surface in front of the vehicle specifically includes the following steps: obtaining road surface information parameters in front of the vehicle, and changing or maintaining the vehicle driving state according to the comparison result of the road surface information parameters and the preset parameters. The road surface information parameters include pits, pit distances, pit directions and shapes, etc. According to the comparison of the road surface information parameters and the preset parameters, if the difference between the road surface information parameters and the preset parameters is within the preset allowable range, the vehicle continues to maintain the driving state, i.e. continues to drive forward at the current speed; if the difference between the road surface information parameters and the preset parameters is not within the preset allowable range, the vehicle driving state is changed, i.e. the vehicle is detoured or parked.

[0062] In this embodiment, the laser radar can also be used to detect the road surface in front of the vehicle, that is, the laser radar can simultaneously detect the obstacles and the road surface in front of the vehicle, so that the production cost of the enterprise can be reduced. When the laser radar detects the road surface in front of the vehicle, the image shown in Figure 3 is first obtained, and then three-dimensional modeling is performed to obtain the image shown in Figure 4 after point cloud display and gray shading. By comparing Figure 3 and Figure 4 , it can be seen that the pit part of the road surface after three-dimensional modeling is clearer. Of course, other road surface information parameter acquisition devices can also be used to obtain the road surface information parameters in front of the vehicle. The above is only one embodiment of the present application, and should not be limited thereto.

[0063] In some embodiments, step S30 includes the following sub-steps:

[0064] Obtaining an enhanced image of the obstacle in front of the vehicle;

[0065] Extracting multi-level depth features of the enhanced image;

[0066] Detecting and analyzing the multi-level depth features to obtain a detection and analysis result.

[0067] The enhanced image of the obstacle in front of the vehicle is obtained by photographing the obstacle in front of the vehicle through a multispectral camera; the multi-level deep features of the enhanced image are extracted by using a YOLO model with a residual neural network and a convolutional neural network, the YOLO model with the residual neural network and the convolutional neural network extracts the object features and extracts the object position from the network at the same time, that is, positioning and classification are realized in the same convolutional neural network, so that the category probability and the coordinates of the obstacle are directly obtained.

[0068] Preferably, after the multi-level deep features of the enhanced image are extracted, the multi-level deep features are fused to obtain semantic features, and the semantic features are detected and analyzed. Since the effective information of the picture decreases with the increase of the level when the multi-level deep features are extracted, the semantic features obtained by fusing the multi-level deep features contain the deep features of all levels, which improves the information richness, especially the information of small objects is more comprehensive, and therefore the accuracy of analyzing the material category of the obstacle is higher.

[0069] Preferably, before the enhanced image of the obstacle in front of the vehicle is obtained, the position of the device for detecting and analyzing the obstacle in front of the vehicle can be adjusted by the adjusting device arranged on the vehicle, that is, the angle of the photographing device is adjusted before photographing the image, so that the photographing device can take a complete and clear photograph of the obstacle in front of the vehicle.

[0070] Next, taking the Xiangtan Heavy Industry MCC600D heavy mining dump truck with a self-weight of 180 tons and a total weight of 600 tons under full load as an example, the automatic driving obstacle detection and recognition method of the present application is specifically described.

[0071] The Xiangtan Heavy Industries MCC600D heavy mine dump truck travels at a speed of 50 km / h under full load. First, the millimeter wave radar detects whether there is an obstacle in front of the vehicle at a long distance (300 meters). After detecting an obstacle in front of the vehicle, the vehicle is controlled to decelerate for the first time, and the speed is reduced to 25 km / h-30 km / h. Second, the laser radar collects point cloud data of the obstacle in front of the vehicle at a medium distance (100-150 meters). The laser radar point cloud is fused with the millimeter wave radar point cloud. According to the fusion result, the volume of the obstacle is identified. If the volume of the obstacle is greater than a preset volume (such as 0.5m*0.5m*0.5m), the vehicle is decelerated for the second time, and the speed is reduced to 5 km / h-10 km / h. Otherwise, the second deceleration is not performed. Finally, while the vehicle is decelerated for the second time, the scanning angle of the multispectral camera is adjusted by the adjusting device. The material of the obstacle at a short distance (about 50 meters) is analyzed at a low speed (5 km / h-10 km / h). The vehicle is further controlled according to the result of the spectral analysis. If the material of the obstacle affects the normal driving of the vehicle, the vehicle is controlled to detour or stop. If the material of the obstacle does not affect the normal driving, the vehicle is controlled to continue driving.

[0072] As can be seen from the above, the automatic driving obstacle detection and identification method provided by the embodiments of the present application detects the obstacle in front of the vehicle, decelerates for the first time according to the detection result, collects data of the obstacle in front of the vehicle after the first deceleration, fuses the data collection result with the detection result to model, decelerates for the second time according to the modeling result of the fusion, detects and analyzes the obstacle in front of the vehicle after the second deceleration, and keeps or changes the driving state of the vehicle according to the detection and analysis result. The obstacle in front of the vehicle is detected and identified by hierarchical implementation, so that the vehicle can still detect and identify the obstacle in front of the vehicle under poor observation conditions, and timely adjust the vehicle speed or the vehicle driving state. Not only can the wear caused by sudden braking of the vehicle be reduced, but also the effective obstacle avoidance of the vehicle can be ensured.

[0073] Please refer to Figure 5 , Figure 5 The automatic driving obstacle detection and identification device is integrated in the form of a computer program in the rear-end control device of the vehicle in some embodiments of the present application, and is used for detecting and identifying the obstacle in front of the vehicle. The automatic driving obstacle detection and identification device comprises a detection module 201, a data collection module 202, and a detection and analysis module 203.

[0074] The detection module 201 is configured to detect obstacles in front of the vehicle and perform first deceleration control according to the detection result. The millimeter wave radar is used to detect obstacles in front of the vehicle. The millimeter wave radar determines whether there is an obstacle in front of the vehicle by whether the echo of the emitted millimeter wave is received. In this embodiment, the millimeter wave radar is specifically a 77GHz millimeter wave radar. The 77GHz millimeter wave radar uses 2 transmitters and 74 receivers to realize synthetic aperture imaging of the detection area in front of the vehicle, to obtain high-resolution imaging and realize high-resolution capability in the target azimuth direction. The high-resolution capability in the target distance direction is realized by transmitting a wideband signal. Of course, the millimeter wave radar can also use other devices suitable for long-distance detection. The above is only one embodiment of the present application and should not be limited thereto.

[0075] The data acquisition module 202 is configured to acquire data of obstacles in front of the vehicle after the first deceleration, fuse the data acquisition result with the detection result to model, and perform second deceleration control according to the modeling result. In this embodiment, the laser radar point cloud data and the millimeter wave point cloud data are used to model. Since the millimeter wave radar point cloud is sparse, the point cloud parameters of some feature points of the millimeter wave radar are matched and filtered with the laser radar point cloud to obtain three-dimensional point cloud data. The size of the obstacle is identified according to the three-dimensional point cloud data and compared with a preset value. If the size of the obstacle is smaller than the preset value, the vehicle does not need to decelerate. If the size of the obstacle is larger than the preset value, the vehicle is controlled to decelerate. Specifically, the laser radar is a 128-line laser radar.

[0076] The detection analysis module 203 is configured to detect and analyze the obstacle in front of the vehicle after the second deceleration, and keep or change the vehicle driving state according to the detection analysis result. Keeping the vehicle driving state means that the vehicle continues to drive forward at the current speed. Changing the vehicle driving state means that the vehicle performs detouring or parking. If the material of the obstacle affects the normal driving of the vehicle, the vehicle driving state is changed, that is, a detouring or parking instruction is generated to control the vehicle to perform detouring or parking. If the material of the obstacle does not affect the normal driving of the vehicle, the vehicle driving state is kept, that is, a continue-to-drive-forward instruction is generated to control the vehicle to continue to drive forward at the current speed.

[0077] In some embodiments, the data acquisition module 202 is configured to perform the following steps when performing second deceleration control according to the modeling result: determining whether to perform second deceleration control according to whether the modeling result is within a preset value range. If the modeling result is within the preset value range, the vehicle does not need to decelerate. If the modeling result is not within the preset value range, the vehicle is controlled to decelerate.

[0078] In some embodiments, the data acquisition module 202 is configured to perform the following steps when fusing the data acquisition result with the detection result: obtaining geographical reference information of the vehicle and relative information of the working environment of the vehicle relative to the laser scanner coordinate system; performing coordinate conversion on the geographical reference information and the relative information to obtain three-dimensional point cloud information.

[0079] The geographical reference information of the vehicle can be directly obtained by the IMU / GNSS system, and the IMU / GNSS system obtains point cloud data; the relative information of the working environment of the vehicle relative to the laser scanner coordinate system can be collected by the laser radar, and the laser radar collects point cloud data; the point cloud data of each frame of the geographical reference information and the relative information is converted from the IMU / GNSS coordinate system and the laser scanner coordinate system to the geocentric rectangular coordinate system by an interpolation algorithm, so as to realize coordinate conversion of the geographical reference information and the relative information.

[0080] Specifically, the coordinate conversion of the geographical reference information and the relative information is calculated by the following formula:

[0081] ;

[0082] wherein, is the coordinate of the laser scanning point P in the geocentric rectangular coordinate system; is the coordinate of the IMU / GNSS center in the geocentric rectangular coordinate system, i.e., the geographical reference information of the vehicle, which is obtained by the position output of the IMU / GNSS system; is the rotation matrix of the IMU / GNSS coordinate system to the geocentric rectangular coordinate system, which is formed by the attitude measured by the IMU / GNSS system; is the component expression of the bias of the scanning center of the laser scanner to the IMU / GNSS center in the IMU / GNSS coordinate system, and the initial value can be obtained by manual measurement; is the rotation matrix of the laser scanner coordinate system to the IMU / GNSS coordinate system, which is determined by the specific installation axis, i.e., which is determined by the specific installation position of the laser scanner; is the coordinate of the laser scanning point in the laser scanner coordinate system, i.e., the relative information of the working environment of the vehicle relative to the laser scanner coordinate system, which is output by the laser scanner.

[0083] In some embodiments, the detection analysis module 203 is configured to perform the following steps when detecting and analyzing the obstacle in front of the vehicle after the second deceleration: obtaining an enhanced image of the obstacle in front of the vehicle; extracting multi-level depth features of the enhanced image; and performing detection analysis on the multi-level depth features to obtain a detection analysis result.

[0084] In some embodiments, the detection analysis module 203 is configured to further perform the following steps after extracting the multi-level deep features of the enhanced image and before performing the detection analysis on the multi-level deep features: fusing the multi-level deep features to obtain semantic features. Fusing the multi-level deep features reduces the size of the feature map and makes the information contained more rich, thereby improving the accuracy of the detection analysis.

[0085] In some embodiments, the automatic driving obstacle detection and recognition device further comprises an adjustment module. The adjustment module is configured to adjust the position of the device for detecting and analyzing the obstacles in front of the vehicle before performing the detection analysis on the obstacles in front of the vehicle after the second deceleration. Specifically, the adjustment module can be a mechanical hand or other device that can adjust the angle or displacement of the acquisition module.

[0086] As can be seen from the above, the automatic driving obstacle detection and recognition device provided by the embodiments of the present application detects the obstacles in front of the vehicle, performs the first deceleration control according to the detection result, collects data of the obstacles in front of the vehicle after the first deceleration, fuses the data collection result with the detection result to model, performs the second deceleration control according to the fusion modeling result, performs the detection analysis on the obstacles in front of the vehicle after the second deceleration, and keeps or changes the driving state of the vehicle according to the detection analysis result. The obstacles in front of the vehicle are detected and recognized through hierarchical levels, so that the vehicle can still detect and recognize the obstacles in front of the vehicle in the case of poor observation conditions, and timely adjust the vehicle speed or the driving state of the vehicle. Not only can the wear caused by sudden braking of the vehicle be reduced, but also the effective obstacle avoidance of the vehicle can be ensured.

[0087] Please refer to Figure 6 , Figure 6 A structure schematic diagram of an electronic device provided by the embodiments of the present application is provided. The present application provides an electronic device 3, comprising a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanism (not shown). The memory 302 stores a computer program executable by the processor 301. When the computing device is running, the processor 301 executes the computer program to execute the method in any optional implementation manner of the above-mentioned embodiments to realize the following functions: detecting the obstacles in front of the vehicle, performing the first deceleration control according to the detection result; collecting data of the obstacles in front of the vehicle after the first deceleration, fusing the data collection result with the detection result to model, performing the second deceleration control according to the fusion modeling result; performing the detection analysis on the obstacles in front of the vehicle after the second deceleration, and keeping or changing the driving state of the vehicle according to the detection analysis result.

[0088] The embodiment of the present application provides a storage medium, which stores a computer program, and the computer program is executed by a processor to execute the method in any optional implementation manner of the above embodiment, so as to realize the following functions: obstacle detection is performed on the front of a vehicle, first deceleration control is performed according to a detection result; data collection is performed on the obstacle in front of the vehicle after the first deceleration, a fusion modeling is performed on the data collection result and the detection result, second deceleration control is performed according to a fusion modeling result; detection analysis is performed on the obstacle in front of the vehicle after the second deceleration, and a vehicle driving state is kept or changed according to a detection analysis result. The storage medium can be realized by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

[0089] In the embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented by other ways. The device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there can be another division way in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0090] In addition, the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment.

[0091] Furthermore, the function modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0092] In this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.

[0093] The above description is merely illustrative of the application and not intended to be limiting. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the application and are included within its spirit and scope. Any modification, equivalent replacement or improvement made without departing from the spirit and principle of the application should be included in the scope of protection of the application.

Claims

1. An obstacle detection and recognition method for autonomous driving, used to detect and recognize obstacles in front of a vehicle, characterized in that, The method includes the following steps: The vehicle uses millimeter-wave radar to detect obstacles in front of it from a distance, and triggers the first deceleration control when an obstacle is detected. After the first deceleration, point cloud data of obstacles in front of the vehicle is collected by lidar. The lidar point cloud data is fused with the millimeter-wave radar detection results to form a model. When the fused model identifies that the volume of the obstacle is greater than a preset threshold, the second deceleration control is triggered. After the second deceleration, the obstacles in front of the vehicle are detected and analyzed, and the vehicle's driving state is maintained or changed based on the detection and analysis results. The detection and analysis of obstacles in front of the vehicle after the second deceleration includes the following steps: Acquire an enhanced image of the obstacle in front of the vehicle; Extract multi-level depth features from the enhanced image; The multi-level deep features are detected and analyzed.

2. The autonomous driving obstacle detection and recognition method according to claim 1, characterized in that, The method further includes the following steps after extracting the multi-level depth features of the enhanced image and before detecting and analyzing the multi-level depth features: The multi-level deep features are fused to obtain semantic features.

3. The autonomous driving obstacle detection and recognition method according to claim 1, characterized in that, The process of fusing and modeling the lidar point cloud data with the millimeter-wave radar detection results includes the following steps: Obtain the geographic reference information of the vehicle and the relative information of the vehicle's working environment with respect to the coordinate system of the laser scanner; Coordinate transformation is performed on the geographic reference information and the relative information.

4. The autonomous driving obstacle detection and recognition method according to claim 3, characterized in that, The coordinate transformation of the geographic reference information and the relative information is calculated using the following formula: in, Let P be the coordinates of the laser scanning point P in the geocentric rectangular coordinate system; The coordinates of the IMU / GNSS center in the geocentric rectangular coordinate system are obtained from the position output measured by the IMU / GNSS system. The rotation matrix from the IMU / GNSS coordinate system to the geocentric rectangular coordinate system is formed by the attitude measured by the IMU / GNSS system; The component of the offset from the laser scanner center to the IMU / GNSS center in the IMU / GNSS coordinate system is expressed, and the initial value is obtained through manual measurement. The rotation matrix from the laser scanner coordinate system to the IMU / GNSS coordinate system is determined by the specific installation axis; These are the coordinates of the laser scanner's scan point in the laser scanner's coordinate system, output by the laser scanner.

5. The autonomous driving obstacle detection and recognition method according to claim 1, characterized in that, Before detecting and analyzing obstacles in front of the vehicle after the second deceleration, the method further includes the following steps: adjusting the position of the device for detecting and analyzing obstacles in front of the vehicle.

6. An obstacle detection and recognition device for autonomous driving, used to detect and recognize obstacles in front of a vehicle, characterized in that, The device includes: The detection module is used to detect obstacles in front of the vehicle at a long distance using millimeter-wave radar, and triggers the first deceleration control when an obstacle is detected. The data acquisition module is used to collect point cloud data of obstacles in front of the vehicle using LiDAR after the first deceleration, and to fuse the LiDAR point cloud data with the detection results of the millimeter-wave radar to form a model. When the fused model identifies that the volume of the obstacle is greater than a preset threshold, the second deceleration control is triggered. The detection and analysis module is used to detect and analyze obstacles in front of the vehicle after the second deceleration, and maintain or change the vehicle's driving state based on the detection and analysis results. The detection and analysis of obstacles in front of the vehicle after the second deceleration includes the following steps: Acquire an enhanced image of the obstacle in front of the vehicle; Extract multi-level depth features from the enhanced image; The multi-level deep features are detected and analyzed.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer-readable instructions that, when executed by the processor, perform the steps of the method as described in any one of claims 1-5.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the steps of the method as described in any one of claims 1-5.

Images