Obstacle detection method, obstacle detection device applied to a vehicle

Abstract

The present invention provides an obstacle detection method and an obstacle detection device applied to a vehicle. The method includes: determining target point cloud data according to the point cloud data obtained by a radar sensor and the image information collected by an image sensor; respectively performing obstacle recognition on the target point cloud data and the image information to obtain first obstacle information corresponding to the target point cloud data and second obstacle information corresponding to the image information; performing obstacle fusion on the first obstacle information and the second obstacle information to determine a target obstacle. The present invention can improve the detection accuracy of the target obstacle.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving, and in particular, to an obstacle detection method and an obstacle detection device applied to a vehicle. Background Art

[0002] Autonomous vehicles can provide higher safety, productivity, and traffic flow, and will play an important role in future urban transportation systems. In most autonomous driving scenarios or assisted driving scenarios, surrounding environment perception is a crucial task, and single sensors have different disadvantages in environment perception. Therefore, multi-sensor fusion has become a necessary means to improve the effect of the perception system.

[0003] Currently, a multi-sensor fusion method is generally used for obstacle detection, that is, an obstacle detection method based on data-level fusion. The obstacle detection method based on data-level fusion is to transmit all the original data to a processor for data processing to determine obstacles.

[0004] However, the above-mentioned obstacle detection method based on data-level fusion has the problem of low obstacle detection accuracy. Summary of the Invention

[0005] Embodiments of the present invention provide an obstacle detection method and an obstacle detection device applied to a vehicle to solve the problem of low obstacle detection accuracy in the existing detection method.

[0006] In a first aspect, embodiments of the present invention provide an obstacle detection method, including:

[0007] Determine target point cloud data according to the point cloud data obtained by a radar sensor and the image information collected by an image sensor;

[0008] Respectively perform obstacle recognition on the target point cloud data and the image information to obtain first obstacle information corresponding to the target point cloud data and second obstacle information corresponding to the image information;

[0009] Perform obstacle fusion on the first obstacle information and the second obstacle information to determine a target obstacle.

[0010] In a second aspect, embodiments of the present invention provide an obstacle detection device applied to a vehicle, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method in the first aspect or any possible implementation manner of the first aspect are implemented.

[0011] In a third aspect, embodiments of the present invention provide an obstacle detection device, including:

[0012] A point cloud data determination module, configured to determine target point cloud data according to the point cloud data obtained by a radar sensor and the image information collected by an image sensor;

[0013] An obstacle information determination module, configured to perform obstacle recognition on the target point cloud data and the image information respectively, to obtain first obstacle information corresponding to the target point cloud data and second obstacle information corresponding to the image information;

[0014] A target obstacle determination module, configured to perform obstacle fusion on the first obstacle information and the second obstacle information to determine a target obstacle.

[0015] Fourthly, an embodiment of the present invention provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method in the first aspect or any possible implementation manner of the first aspect are implemented.

[0016] An embodiment of the present invention provides an obstacle detection method and an obstacle detection device applied to a vehicle. After obtaining point cloud data through a radar sensor and collecting a target image through an image sensor, the target image and the point cloud data are fused at the data level to determine target point cloud data. Then, obstacle recognition is performed on the target point cloud data to obtain one piece of obstacle information, and at the same time, obstacle recognition is performed on the target image to obtain another piece of obstacle information. After that, the first obstacle information and the second obstacle information are fused at the target level to jointly determine a target obstacle. Since the target image collected by the image sensor not only assists the radar sensor in obstacle recognition at the target level, but also can assist the radar sensor in judging the point cloud data at the data level to determine the target point cloud data, and perform obstacle recognition based on the target point cloud data, the obtained detection result is more accurate, and the detection accuracy of the target obstacle is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is a centralized fusion structure diagram provided by an embodiment of the present invention;

[0019] Figure 2 It is a distributed fusion structure diagram provided by an embodiment of the present invention;

[0020] Figure 3 It is a hybrid fusion structure diagram provided by an embodiment of the present invention;

[0021] Figure 4 It is a flowchart showing the implementation of an obstacle detection method provided by an embodiment of the present invention;

[0022] Figure 5 It is an improved hybrid fusion structure diagram provided by an embodiment of the present invention;

[0023] Figure 6 It is a schematic diagram showing the positional relationship among a radar, a vehicle, and a camera provided by an embodiment of the present invention;

[0024] Figure 7 It is a flowchart showing the implementation of an obstacle detection method provided by another embodiment of the present invention;

[0025] Figure 8 It is a schematic diagram showing the geometric relationship between data points, a vehicle coordinate system, and a sensor coordinate system provided by an embodiment of the present invention;

[0026] Figure 9 It is a schematic diagram showing the geometric relationship of vehicle turning provided by an embodiment of the present invention;

[0027] Figure 10 It is a schematic diagram showing data time synchronization provided by an embodiment of the present invention;

[0028] Figure 11 It is a schematic diagram of a forward radar coordinate system provided by an embodiment of the present invention;

[0029] Figure 12 It is a schematic diagram showing the positional relationship among an image coordinate system, a camera coordinate system, and a vehicle coordinate system provided by an embodiment of the present invention;

[0030] Figure 13 It is a schematic diagram showing the positional relationship between an image coordinate system and a pixel coordinate system provided by an embodiment of the present invention;

[0031] Figure 14 It is a flowchart showing the implementation of obstacle fusion provided by an embodiment of the present invention;

[0032] Figure 15 It is a schematic diagram of the structure of an obstacle detection device provided by an embodiment of the present invention;

[0033] Figure 16 It is a schematic diagram of the structure of an obstacle detection device applied to a vehicle provided by an embodiment of the present invention. Detailed implementation manners

[0034] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present invention. However, those skilled in the art should clearly understand that the present invention can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present invention.

[0035] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will be described through specific embodiments with reference to the accompanying drawings.

[0036] Autonomous vehicles can provide higher safety, productivity, and traffic efficiency and will play an important role in future urban transportation systems. In most autonomous driving scenarios or assisted driving scenarios, surrounding environment perception is a crucial task. Single sensors (such as lidar, millimeter-wave radar, cameras, and ultrasonic sensors) have different disadvantages in environment perception. Therefore, multi-sensor fusion has become a necessary means to improve the effect of environment perception.

[0037] According to the degree of data processing by local sensors in multi-sensor fusion, the fusion methods of multi-sensors can be classified, mainly including centralized, distributed, and hybrid. Combining Figures 1 - 3 The above three fusion methods are described as follows:

[0038] Figure 1 is the centralized fusion structure diagram. Through Figure 1 it can be seen that in the centralized method, all information of the sensors is sent to the domain controller, and data association, measurement fusion, and target tracking are carried out in sequence. Finally, the position and status information of the target are obtained, and then a decision is made. The advantage of the centralized method is high data processing accuracy, while the disadvantage is that a large amount of data is likely to cause excessive communication load and high requirements for the processing performance of the controller. Figure 2 is the distributed fusion structure diagram. Through Figure 2 it can be seen that in the distributed method, the target observation results of each sensor are locally processed for relevant target detection and tracking, and then sent to the domain controller to obtain the local track information of multi-target tracking. The advantages of the distributed method are low demand for communication bandwidth and fast calculation speed, while the disadvantage is that the tracking accuracy is much lower than that of the centralized method. Figure 3 is the hybrid fusion structure diagram. Through Figure 3 it can be seen that the hybrid method forms a hybrid structure according to different requirements for sensor data, combining the advantages of the centralized and distributed structures and making up for their deficiencies.

[0039] However, Figure 3In the shown hybrid fusion structure, the fusion of different types of sensors, such as the fusion of an image sensor (camera) and a radar sensor (corner radar, forward radar), mainly focuses on the target level, and the detection accuracy of obstacles is relatively low. Among them, 5R1V refers to the sensor configuration of 5 millimeter-wave radars and 1 forward multi-functional camera. Based on the Figure 3 shown hybrid fusion structure, in order to further improve the obstacle detection accuracy, an improved hybrid fusion structure and an obstacle detection method are proposed.

[0040] Refer to Figure 4 and Figure 5 , Figure 4 which is the implementation flowchart of an obstacle detection method provided by an embodiment of the present invention, and it is applicable to the Figure 5 shown improved hybrid fusion structure. Figure 5 In the shown improved hybrid fusion structure, the sensors used for obstacle detection include a radar sensor and an image sensor. Among them, the radar sensor further includes a forward radar sensor and a lateral radar sensor, and the number of lateral radar sensors can be multiple, such as 2, 4 or more. Preferably, the number of lateral radar sensors is 4. At this time, the Figure 5 shown hybrid fusion structure can be called a 5R1V fusion structure. Among them, "5R" refers to 5 radar sensors, that is, 1 forward radar sensor and 4 lateral radar sensors, and "1V" refers to 1 image sensor. This structure is mainly aimed at the high-speed driving scenario for L3 level, and more accurate and reliable obstacle detection results can be obtained. In some embodiments, the radar sensor can adopt a millimeter-wave radar sensor, and the image sensor is a camera.

[0041] The following will take the Figure 5 shown hybrid fusion structure as an example to introduce the specific process of the obstacle detection method of each embodiment of the present invention. It includes the following steps:

[0042] Step S101: Determine the target point cloud data according to the point cloud data obtained by the radar sensor and the image information collected by the image sensor;

[0043] Step S102: Respectively perform obstacle recognition on the target point cloud data and the image information to obtain the first obstacle information corresponding to the target point cloud data and the second obstacle information corresponding to the image information;

[0044] Step S103: Perform obstacle fusion on the first obstacle information and the second obstacle information to determine the target obstacle.

[0045] Specifically, Figure 6Shows the positional relationship among the radar, camera, and vehicle. Among them, 1 forward radar sensor and 1 camera are both installed at the front end of the vehicle, and 4 high-resolution side radar sensors (corner radars) are installed on the 4 sides of the vehicle. Combining Figure 6 , the specific process of the present invention is as follows: During the driving of the vehicle, the forward radar sensor collects point cloud data of the corresponding detection area, the 4 side radar sensors respectively collect point cloud data of the corresponding detection areas, the camera collects image information within the shooting range. Through the point cloud data collected by the above radar sensors and the image information collected by the camera, target point cloud data is obtained. Then, the target point cloud data is used for obstacle recognition to determine the first obstacle information, and the image information is used for obstacle recognition to determine the second obstacle information. Finally, the first obstacle information and the second obstacle information are used for obstacle fusion to obtain the target obstacle. In addition, the steps of determining the first obstacle and determining the second obstacle are not limited in order, and they can be executed simultaneously.

[0046] Compared with the prior art, for the obstacle detection method provided by the embodiment of the present invention, after obtaining the point cloud data through the radar sensor and collecting the target image through the image sensor, the target image and the point cloud data are fused at the data level to determine the target point cloud data. Then, the target point cloud data is used for obstacle recognition to obtain an obstacle information (i.e., the first obstacle information), and at the same time, the target image is used for obstacle recognition to obtain another obstacle information (i.e., the second obstacle information). Then, the first obstacle information and the second obstacle information are fused at the target level to jointly determine the target obstacle. Since the target image collected by the image sensor not only assists the radar sensor in obstacle recognition at the target level, but also can assist the radar sensor in judging the point cloud data at the data level to determine the target point cloud data and perform obstacle recognition based on the target point cloud data, the obtained detection result is more accurate, and the detection accuracy of the target obstacle is improved.

[0047] In one embodiment, the above step S101 may include the following steps:

[0048] Step S201: Synchronize the point cloud data and the image information to obtain the synchronized point cloud data and the synchronized image information;

[0049] Among them, the synchronization process includes time synchronization processing and space synchronization processing.

[0050] Step S202: Grid the synchronized point cloud data to obtain the first grid map, and grid the synchronized image information to obtain the second grid map;

[0051] Step S203: Fuse the first grid map and the first grid map according to a preset fusion method to obtain a fused grid map;

[0052] Step S204: Based on the fused raster map, perform attribute correction on the point cloud data to obtain the target point cloud data.

[0053] Specifically, taking Figure 5 the 5R1V hybrid fusion structure shown as an example, the specific implementation process of the obstacle detection method is described. That is, the radar sensor includes a forward radar sensor and a lateral radar sensor, and when the radar sensor is a millimeter-wave radar sensor.

[0054] When performing obstacle detection through multiple sensors, the data of multiple sensors need to meet the requirements of time synchronization and space synchronization. Therefore, before processing the data of the sensors, it is first necessary to perform synchronization processing on the data of each sensor.

[0055] Figure 10 This is a schematic diagram of time synchronization for the data collected by the radar sensor and the image sensor in an embodiment of the present invention. The present invention performs unified GPS timing for the radar sensor and the image sensor. After unified timing, the radar sensor and the image sensor perform time synchronization according to Lagrange interpolation. From Figure 10 it can be seen that the data collected by each sensor has a GPS timestamp. The GPS timestamp of the radar sensor can be considered as the time when the domain controller obtains the point cloud data collected by the radar sensor within the current reporting period, and the GPS timestamp of the image sensor can be considered as the time when the domain controller obtains the image information collected by the image sensor within the current reporting period. When each sensor has a corresponding GPS timestamp, Lagrange interpolation is used to perform time synchronization on each sensor. The process of performing time synchronization on each sensor belongs to common knowledge and will not be elaborated here.

[0056] After time synchronization, further perform space synchronization on the data of each sensor after time synchronization. Space synchronization mainly maps the data collected by each sensor into a unified coordinate system. In this embodiment, this unified coordinate system is a coordinate system centered on the rear axle of the vehicle (hereinafter uniformly described as the vehicle coordinate system). For different sensors, the process of their space synchronization is as follows:

[0057] (1) For the forward millimeter-wave radar, based on the conversion relationship between the forward millimeter-wave radar coordinate system and the vehicle coordinate system, convert the point cloud data corresponding to the forward millimeter-wave radar into the vehicle coordinate system to obtain the synchronized data corresponding to the forward millimeter-wave radar.

[0058] Forward millimeter-wave radar coordinate system X R Y R Z R -O R As Figure 11 shown, define the installation position of the forward millimeter-wave radar as the coordinate origin OR , the directions of the three coordinate axes are the same as those of the vehicle coordinate system. The detection direction of the forward millimeter-wave radar is the X-axis direction, and X R O R Y R is the detection plane of the forward millimeter-wave radar, and Y R O R Z R is the installation plane. The target data output by the forward millimeter-wave radar includes distance, vehicle speed, relative angle, etc., which is two-dimensional information in the X R O R Y R plane. The Y R O R Z R plane is parallel to the Y W O W Z W plane, and the distance is X0. The X R O R Y R plane is parallel to the X W O W Y W plane, and the distance is H. For the forward millimeter-wave radar target P(R,α), the conversion relationship between the forward millimeter-wave radar coordinate system and the vehicle coordinate system is:

[0059]

[0060] where R represents the target distance and α represents the azimuth angle.

[0061] (2) For the side millimeter-wave radar, based on the conversion relationship between the side millimeter-wave radar coordinate system and the vehicle coordinate system, the point cloud data corresponding to the side millimeter-wave radar is converted into the vehicle coordinate system to obtain the synchronized data corresponding to the side millimeter-wave radar.

[0062] The side millimeter-wave radar coordinate system X Ri Y Ri Z Ri -O Ri , i = 1, 2, 3, 4 and the conversion relationship with the vehicle coordinate system is:

[0063]

[0064] where (X i , Y i , Z i ) is the installation position of the side millimeter-wave radar in the vehicle coordinate system, θ Pi , the azimuth angle and pitch angle of the target detected by the i-th side millimeter-wave radar, ω i , φ iThe azimuth and elevation angles for the installation of the side-mounted millimeter-wave radar.

[0065] (3) For the image sensor, taking the camera as an example, when performing spatial synchronization, it is mainly based on the conversion relationships between the image coordinate system and the pixel coordinate system, the camera coordinate system and the image coordinate system, the world coordinate system and the camera coordinate system, and the world coordinate system and the pixel coordinate system. The time-synchronized image information is converted into the vehicle coordinate system to obtain the spatially synchronized image information.

[0066] The specific implementation process of spatially synchronizing the image information can be referred to Figure 12 and Figure 13 , Figure 12 is a schematic diagram of the positional relationship among the image coordinate system, the camera coordinate system, and the vehicle coordinate system, Figure 13 is a schematic diagram of the positional relationship between the image coordinate system and the pixel coordinate system. Based on the linear camera model, first, through the conversion relationship between the camera coordinate system and the image coordinate system, the coordinates of each point in the image are determined. Then, through the conversion relationships between the image coordinate system and the pixel coordinate system, and the pixel coordinate system and the world coordinate system in sequence, the corresponding coordinates of each point in the image projected into the world coordinate system are obtained. Among them, the world coordinate system is the vehicle coordinate system. Through the above conversion process, the conversion between the vehicle coordinate system and the camera coordinate system is completed to achieve the three-dimensional reconstruction of the point coordinates in the planar image.

[0067] Image coordinate system xoy: The coordinate system of the imaging plane after the camera projects the objects in the three-dimensional real environment through perspective projection. Define the intersection point of the optical axis and the imaging plane as the coordinate origin O, and the imaging plane as the coordinate system plane. The image information stored in the computer is based on the pixel coordinate system, which defines the upper left vertex of the image as the origin of the pixel coordinate system uO0v, as Figure 13 shown. The origin O of the image coordinate system is located at the pixel point (u0, v0) in the pixel coordinate system. Then, the conversion relationship between the image coordinate system and the pixel coordinate system is:

[0068]

[0069] where dx and dy respectively represent the physical sizes of each pixel point in the x and y directions of the image coordinate system.

[0070] Camera coordinate system X c Y c Z c -O c : A coordinate system with the center of the camera's optical lens as the origin O c , and the optical axis of the camera as the Z c axis. Its coordinate axes are parallel to the image coordinate axes. Then, the conversion relationship between the camera coordinate system and the image coordinate system (f is the focal length of the camera) is:

[0071]

[0072] World coordinate system (X W Y W Z W )): As a reference coordinate system, it is used to describe the installation positions of the radar and the camera (i.e., the camera in this application), as well as the positions of other objects in space. The conversion relationship between the world coordinate system and the camera coordinate system is:

[0073]

[0074] Among them, the rotation matrix R is a 3×3 unitary orthogonal matrix, representing the rotation relationship of the camera coordinate system relative to the world coordinate system. The translation vector T c is a vector suitable for describing the translation relationship of the camera coordinate system relative to the world coordinate system.

[0075] Finally, the conversion relationship between the world coordinate system and the pixel coordinate system is obtained as:

[0076]

[0077]

[0078] In the formula, M1 is the camera internal parameter matrix, and M2 is the camera external parameter matrix.

[0079] After synchronizing the data of each sensor in time and space, it is necessary to fuse the data of each sensor at the data level. When performing data fusion, first, rasterize the synchronized point cloud data to obtain a first raster map, and rasterize the synchronized image information to obtain a second raster map. Then, fuse the first raster map and the first raster map according to a preset fusion method to obtain a fused raster map, and correct the attributes of the point cloud data based on the fused raster map to obtain target point cloud data, thereby realizing the process of correcting the point cloud data using image information, that is, the process of executing the above steps S202 to S204.

[0080] Among them, before rasterizing the point cloud data, it is necessary to separate the moving and static parts of the point cloud data to obtain dynamic point cloud data and static point cloud data. Among them, separating the moving and static parts of the point cloud data is mainly achieved based on the vehicle projection speed and the point cloud Doppler speed. Specifically, the realization of separating the moving and static parts includes the following steps:

[0081] (11) Obtain the measured Doppler speed of each data point in the point cloud data;

[0082] (12) Calculate the target Doppler speed of each data point according to the current vehicle speed;

[0083] (13) calculating the difference between the measured Doppler velocity and the target Doppler velocity;

[0084] (14) When the absolute value of the difference is greater than a preset threshold, the data point is marked as a moving point. When the absolute value of the difference is less than or equal to the preset threshold, the data point is marked as a static point. Thus, the point cloud data is divided into dynamic point cloud data corresponding to the moving point and static point cloud data corresponding to the static point.

[0085] In the above implementation process, the measured Doppler velocity is the velocity information carried in the point cloud data, and the target Doppler velocity is related to the current vehicle speed. Depending on the vehicle driving state, the vehicle speed is represented in different ways. In this embodiment, the vehicle driving state includes: a straight driving state and a non-straight driving state. Based on different driving states, the current vehicle speed is represented in different ways, so the target Doppler velocity is calculated in different ways. The following describes the process of performing dynamic and static separation under different vehicle driving states.

[0086] (1) When the vehicle is in a straight-line driving state, the vehicle speed is projected to the direction of the line connecting the point and the radar center, that is, the point cloud Doppler can be expressed as: V di =V ego cosθ i , where V di V is the point cloud Doppler velocity inferred from the vehicle speed, i.e., the target Doppler velocity; ego is the vehicle speed, θ i is the sum of the azimuth and installation angle of the i-th point cloud, combined with Figure 8 , taking the right front radar sensor in the radar sensor as an example, the installation angle α is the angle between the connecting line of the vehicle coordinate system origin E and the right front radar sensor coordinate system origin S and the vehicle coordinate system Y axis, and the azimuth angle γ is the angle passed by the normal vector y axis of the right front radar sensor coordinate system origin S when the connecting line of the i-th data point and the right front radar sensor coordinate system origin S rotates along the minimum path, wherein the azimuth angle γ is positive when rotating counterclockwise, and negative when rotating clockwise; then the difference between the Doller velocity and the target is calculated, namely: If the speed difference is greater than the preset threshold, it is a moving point, otherwise it is a static point, where is the point cloud Doppler velocity actually measured by the radar.

[0087] (2) When the vehicle is not traveling in a straight line, V diThe calculation is determined by the linear velocity of the vehicle. The formula for the linear velocity of the vehicle is: V = ωR, where ω is the angular velocity (yawRate), R is the turning radius, and V is the linear velocity. Since the turning radii of the left and right wheels are different, with a difference of one wheelbase, the linear velocities of the left and right wheels are different during turning, and β is the wheel angle. Through engineering experiments, it is found that when the vehicle speed during turning is the inner wheel speed, the error between the measured Doppler value and the theoretical value of the target point is small and stable. Combining Figure 9 we can obtain So V di = V y cosθ i + V x sinθ i where V x is the linear velocity in the X-axis direction of the vehicle coordinate system, and V y is the linear velocity in the Y-axis direction of the vehicle coordinate system.

[0088] Furthermore, since the accuracy of separating moving and static points from the point cloud data directly affects the accuracy of the drivable area detection and the judgment of the attributes of target obstacles, therefore, in the present invention, the point cloud data obtained by each radar sensor is separately separated into moving and static points, that is, the point cloud data obtained by the forward millimeter-wave radar and the point cloud data obtained by the side millimeter-wave radar are separately separated into moving and static points, obtaining the dynamic point cloud data and static point cloud data corresponding to the forward millimeter-wave radar, and the dynamic point cloud data and static point cloud data corresponding to the side millimeter-wave radar. In one embodiment, the step of separating moving and static points from the point cloud data is performed before the synchronization process, that is, during the synchronization process, the dynamic point cloud data and static point cloud data of each obtained radar sensor are synchronized.

[0089] In step S202, the synchronized point cloud data is rasterized to obtain a first raster map, and the synchronized image information is rasterized to obtain a second raster map.

[0090] Among them, the process of rasterizing the point cloud data obtained by each radar sensor may include the following steps: rasterize the detection area of the radar sensor, count the number of static points in each grid. When the number of static points contained in the grid is greater than the first target threshold, the grid is an occupied grid, and the attribute of the grid is marked as occupied. When the number of static points contained in the grid is less than or equal to the first target threshold, the grid is an invalid grid, and the attribute of the grid is marked as invalid, thereby obtaining a first raster map composed of occupied grids and invalid grids. In this process, the process of marking the attributes of each grid is the process of rasterizing the point cloud data.

[0091] When there are multiple radar sensors, the number of the first grid maps is multiple. Further, taking the example that the radar sensors include a forward radar sensor and side radar sensors, at this time, the first grid maps include a forward grid map and side grid maps, where the forward grid map is the first grid map corresponding to the point cloud data obtained by the forward radar sensor, and the side grid maps are the first grid maps corresponding to the point cloud data obtained by the side radar sensors. Further, when the number of side radar sensors is 4, the number of side grid maps is also 4. For the forward radar sensor, the detection area of the forward radar sensor is rasterized, and the number of static points in each grid is counted. When the number of static points contained in a grid is greater than the first target threshold, the grid is an occupied grid; otherwise, the grid is an invalid grid, and the forward grid map is determined by the occupied grids and the invalid grids. Similarly, based on the above operation steps, the grid map corresponding to the side radar sensors, that is, the side grid maps, can be determined, and the specific steps are not elaborated here.

[0092] The process of rasterizing the image information collected by the image sensor includes the following steps:

[0093] The camera acquisition area is rasterized. When there is a non-moving target falling into a certain grid among the multiple grids corresponding to the camera acquisition area, this grid is marked as an occupied grid; otherwise, it is marked as an invalid grid, so as to obtain a second grid map composed of occupied grids and invalid grids. Among them, the non-moving target is a stationary target, such as a stationary vehicle, a bush, or a lamp post, etc. Further, before the step of rasterizing the image information, there is also a target tracking process, that is, the image information collected by the camera is subjected to target tracking to obtain lane lines and target information. Then, the lane lines and the target information are subjected to data time synchronization and data time synchronization in sequence to obtain the synchronized image information.

[0094] Optionally, the above step S203 may include the following steps:

[0095] (1) For each grid, determine the occupancy value of the grid through the occupancy result of the grid in the first grid map and the preset weight of the area to which the grid belongs, as well as the occupancy result of the grid in the second grid map and the preset weight of the area to which the grid belongs;

[0096] (2) Divide the occupancy value of the grid by the number of grid maps to obtain the average occupancy value of the grid;

[0097] (3) When the average occupancy value of the grid is greater than the second target threshold, the grid is an occupied grid; otherwise, the grid is an invalid grid, and the fused grid map is determined by the occupied grids and the invalid grids.

[0098] Specifically, continuing with Figure 5Taking the improved hybrid fusion structure of 5R1V shown as an example, it includes a total of 6 sensors, namely 1 forward millimeter-wave radar, 4 lateral millimeter-wave radars and 1 camera, which includes 5 first grid maps (denoted as 1 forward grid map and 4 lateral grid maps respectively) and 1 second grid map. When performing grid fusion, first, according to the detection accuracy of each sensor, weights weight[i] are set a priori for each sensor in different regions, where i is the sensor number; then, traverse each grid j of the forward grid map, lateral grid map and second grid map, and count the attributes (i.e., occupied or invalid) of each grid in the forward grid map, lateral grid map and second grid map. Based on the statistical results, calculate the target occupancy value cellValue of each grid in the fusion grid map to be obtained. The calculation formula for the target occupancy value is: where N represents the number of sensors, and cell j [i] represents the occupancy value of the j-th grid in the grid map of the i-th sensor. When the grid attribute is occupied, the occupancy value corresponding to the grid is 1, and when the grid attribute is invalid, the occupancy value corresponding to the grid is 0; after that, calculate the average occupancy value. The specific calculation formula is: avgcellValue = cellValue / sensorNum, where sensorNum is the number of sensors; finally, if cellValue is greater than the first preset threshold, the j-th grid of the fused grid map is occupied, otherwise it is invalid, so as to obtain a fused grid map composed of occupied grids and invalid grids. Among them, the process of calculating the attributes of each grid in the fused grid map through the first grid map and the second grid map is the process of data fusion.

[0099] Optionally, the above step S204 may include the following steps:

[0100] (1) When the data point in the point cloud data corresponds to an occupied grid in the fused grid map and the current attribute of the data point is a moving point, correct the attribute of the data point to a static point;

[0101] (2) Use the point cloud data corresponding to the corrected static point and the static point cloud data together as the target point cloud data.

[0102] Specifically, if the data point in the point cloud data falls within the occupied grid in the fused grid map, regardless of what the previous attribute of the data point was, it is set to a static point, where the attributes of the point cloud include static points and moving points.

[0103] Through the above steps S201 to S204, the fusion of the image information collected by the camera and the point cloud data obtained by the radar sensor is realized at the data level, and the process of correcting the point cloud data using the image information is carried out.

[0104] In one embodiment, step S102 includes: performing clustering analysis on the target point cloud data to obtain obstacle point cloud data; performing obstacle tracking on the obstacle point cloud data to obtain obstacle information corresponding to the target point cloud data. Specifically, the present invention fuses the point cloud data obtained by the radar and the image data obtained by the camera, and performs clustering analysis and tracking on the target point cloud determined after fusion, which can effectively improve the recognition accuracy of obstacles.

[0105] In one embodiment, step S103 includes:

[0106] Step S301: Preprocess the first obstacle information to obtain radar track information, and preprocess the second obstacle information to obtain visual track information;

[0107] Step S302: Perform association operations on the radar track information and the visual track information with the fused track information respectively to determine the successfully associated fused track information;

[0108] Step S303: Update the track state of the successfully associated fused track information to obtain the updated fused track information;

[0109] Step S304: Calculate the track confidence of the updated fused track information to obtain the target obstacle.

[0110] Specifically, in combination with Figure 14 , first, the obstacle information corresponding to the target point cloud data and the obstacle information in the image information obtained by the camera are transmitted to the fusion module, and operations such as unified data format adaptation, spatial synchronization, and speed conversion are performed to obtain millimeter-wave radar track information and visual track information. Secondly, the radar track information and the visual track information are respectively associated with the fused track information. Since the characteristics of the radar and vision are different, different association logics or association parameters need to be considered during association. Thirdly, for the fused track information successfully associated with the millimeter-wave radar track information and the visual track information, a Kalman filter update operation is performed, that is, the track state is calculated, that is, information such as the type, motion state, motion mode, and track source of the fused track is updated. Finally, the track confidence of the updated fused track information is calculated.

[0111] In addition, in the track management module, the fused track information is determined by starting with the radar track information or the visual track information. The specific implementation method is as follows: when receiving multiple frames of data, identify whether the first frame of data is radar track information or visual track information. When the first frame of data is radar track information, the radar track information is started as the fused track information; when the first frame of data is visual track information, the visual track information is started as the fused track information.

[0112] It should be understood that the sequence numbers of the steps in the above embodiments do not indicate the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0113] The following is an apparatus embodiment of the present invention. For details not described in detail, reference may be made to the corresponding method embodiments above.

[0114] Figure 15 The structural schematic diagram of an obstacle detection apparatus provided by an embodiment of the present invention is shown. For the sake of convenience of description, only the parts related to the embodiment of the present invention are shown and are described in detail as follows:

[0115] As Figure 15 shown, an obstacle detection apparatus includes:

[0116] A point cloud data determination module 151, configured to determine target point cloud data according to the point cloud data obtained by a radar sensor and the image information collected by an image sensor;

[0117] An obstacle information determination module 152, configured to respectively perform obstacle recognition on the target point cloud data and the image information to obtain first obstacle information corresponding to the target point cloud data and second obstacle information corresponding to the image information;

[0118] A target obstacle determination module 153, configured to perform obstacle fusion on the first obstacle information and the second obstacle information to determine a target obstacle.

[0119] In a possible implementation manner, the target obstacle determination module 153 includes:

[0120] A preprocessing sub-module, configured to respectively preprocess the first obstacle information and the second obstacle information to obtain radar track information and visual track information;

[0121] An association sub-module, configured to respectively perform association operations on the radar track information, the visual track information, and the fusion track information to determine the successfully associated fusion track information, where the fusion track information is determined by starting from the radar track information or the visual track information;

[0122] An update sub-module, configured to update the track state of the successfully associated fusion track information to obtain the updated fusion track information;

[0123] A track estimation sub-module, configured to calculate the track confidence of the updated fusion track information to obtain a target obstacle.

[0124] In a possible implementation manner, the point cloud data determination module 151 includes:

[0125] A synchronization processing sub-module, configured to synchronously process point cloud data and image information to obtain synchronized point cloud data and synchronized image information;

[0126] A rasterization sub-module, configured to rasterize the synchronized point cloud data to obtain a first raster map, and rasterize the synchronized image information to obtain a second raster map;

[0127] A fusion sub-module, configured to fuse the first raster map and the second raster map according to a preset fusion method to obtain a fused raster map;

[0128] A correction sub-module, configured to correct the point cloud data according to the fused raster map to obtain target point cloud data.

[0129] In a possible implementation, before the rasterization sub-module, it further includes:

[0130] A moving and static separation sub-module, configured to separate the point cloud data acquired by a radar sensor into dynamic point cloud data and static point cloud data;

[0131] Correspondingly, the rasterization sub-module includes: rasterizing the detection area of the radar sensor, counting the number of static points in each raster, when the number of static points contained in the raster is greater than a first target threshold, the raster is an occupied raster, otherwise the raster is an invalid raster, and determining the first raster map through the occupied raster and the invalid raster.

[0132] In a possible implementation, the fusion sub-module includes:

[0133] An occupancy value calculation unit, configured to sequentially calculate the target occupancy value of each corresponding raster in the fused raster map according to the weights of the first raster map and the second raster map in the fused raster map, and the occupancy values of each raster in the first raster map and the second raster map;

[0134] An average value calculation unit, configured to divide the target occupancy value by the sum of the quantities of the first raster map and the second raster map to sequentially calculate the average occupancy value of each corresponding raster in the fused raster map;

[0135] A fused raster determination unit, configured to mark the corresponding raster in the fused raster map as an occupied raster when the average occupancy value is greater than a second target threshold, otherwise mark it as an invalid raster, to obtain a fused raster map composed of the occupied raster and the invalid raster.

[0136] In a possible implementation, the correction sub-module includes:

[0137] A judgment unit, configured to correct the attribute of a data point to a static point when the raster corresponding to the data point in the point cloud data is an occupied raster and the current attribute of the data point is a moving point;

[0138] A target point cloud determination unit, configured to use the point cloud data corresponding to the corrected static points and the static point cloud data together as target point cloud data.

[0139] In a possible implementation, the moving and static separation sub-module includes:

[0140] An actual value acquisition unit, configured to acquire the measured Doppler velocity of each data point in the point cloud data;

[0141] A target value calculation unit, configured to calculate the target Doppler velocity of each data point according to the current vehicle speed;

[0142] A difference calculation unit, configured to calculate the difference between the measured Doppler velocity and the target Doppler velocity;

[0143] A judgment unit, configured to mark a data point as a moving point when the absolute value of the difference is greater than a preset threshold, and mark a data point as a static point when the absolute value of the difference is less than or equal to the preset threshold, and the point cloud data is divided into dynamic point cloud data corresponding to the moving points and static point cloud data corresponding to the static points.

[0144] In a possible implementation, the target value calculation unit includes:

[0145] When the vehicle is in a straight-line driving state, calculating the target Doppler velocity of each data point according to the current vehicle speed is specifically:

[0146] V di = V ego cosθ i

[0147] wherein, V di is the target Doppler velocity; V ego is the vehicle driving speed, θ i is the sum of the azimuth angle and the installation angle of the i-th data point, the installation angle is the included angle between the connection line of the origin of the vehicle coordinate system and the origin of the radar sensor or image sensor coordinate system and the Y-axis of the vehicle coordinate system, and the azimuth angle is the angle passed by the connection line of the i-th data point and the origin of the radar sensor or image sensor coordinate system rotating to the normal vector y-axis of the radar sensor or image sensor coordinate system along the minimum path, wherein, when rotating counterclockwise, the azimuth angle is a positive value, and when rotating clockwise, the azimuth angle is a negative value;

[0148] When the vehicle is in a non-straight-line driving state, calculating the target Doppler velocity of each data point according to the current vehicle speed is specifically:

[0149] V di = V y cosθ i + V x sinθi

[0150] Among them, V di is the target Doppler velocity, V x is the linear velocity of the vehicle traveling speed in the X-axis direction of the vehicle coordinate system, V y is the linear velocity of the vehicle traveling speed in the Y-axis direction of the vehicle coordinate system, θ i is the sum of the azimuth angle and the installation angle of the i-th data point.

[0151] In a possible implementation, the radar sensor is a millimeter-wave radar sensor and includes 1 forward radar sensor and 4 side radar sensors, and the image sensor includes 1 camera.

[0152] Figure 16 is a schematic diagram of an obstacle detection device for a vehicle provided by an embodiment of the present invention. As Figure 16 shown, the device 16 of this embodiment includes: a processor 160, a memory 161, and a computer program 162 stored in the memory 161 and executable on the processor 160. When the processor 160 executes the computer program 162, the steps in the above-mentioned various embodiments of the obstacle detection method are implemented, such as Figure 4 the steps 101 to 103 shown. Alternatively, when the processor 160 executes the computer program 162, the functions of each module / unit in the above-mentioned device embodiments are implemented, such as Figure 15 the functions of the modules / units 151 to 153 shown.

[0153] Exemplarily, the computer program 162 can be divided into one or more modules / units, and one or more modules / units are stored in the memory 161 and executed by the processor 160 to complete the present invention. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program 162 in the device 16. For example, the computer program 162 can be divided into Figure 15 the modules / units 151 to 153 shown.

[0154] The device 16 can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The device 16 may include, but is not limited to, a processor 160 and a memory 161. Those skilled in the art can understand that Figure 16 merely an example of the device 16, does not constitute a limitation on the device 16, and may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the terminal may further include input / output devices, network access devices, buses, etc.

[0155] The so-called processor 160 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0156] The memory 161 may be an internal storage unit of the device 16, such as the hard disk or memory of the device 16. The memory 161 may also be an external storage device of the device 16, such as a plug-in hard disk equipped on the device 16, a SmartMedia Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 161 may also include both the internal storage unit of the device 16 and the external storage device. The memory 161 is used to store computer programs and other programs and data required by the terminal. The memory 161 may also be used to temporarily store data that has been output or is to be output.

[0157] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0158] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0159] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0160] In the embodiments provided by the present invention, it should be understood that the disclosed device / terminal and method can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0161] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, the functional units in various embodiments of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0163] When the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of the present invention, it can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described various obstacle detection method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice within the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0164] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention and should all be included in the protection scope of the present invention.

Claims

1. An obstacle detection method, characterized in that, Including: Determine target point cloud data according to the point cloud data obtained by a radar sensor and the image information collected by an image sensor; Perform obstacle recognition on the target point cloud data and the image information respectively to obtain first obstacle information corresponding to the target point cloud data and second obstacle information corresponding to the image information; Perform obstacle fusion on the first obstacle information and the second obstacle information to determine a target obstacle; Performing obstacle fusion on the first obstacle information and the second obstacle information to determine a target obstacle includes: Preprocess the first obstacle information and the second obstacle information respectively to obtain radar track information and visual track information; Perform association operations on the radar track information, the visual track information, and the fusion track information respectively to determine the successfully associated fusion track information, where the fusion track information is determined by starting from the radar track information or the visual track information; Update the track state of the successfully associated fusion track information to obtain updated fusion track information; Calculate the track confidence of the updated fusion track information to obtain the target obstacle.

2. The method according to claim 1, characterized in that, The determining the target point cloud data according to the point cloud data obtained by the radar sensor and the image information collected by the image sensor includes: Synchronize the point cloud data and the image information to obtain synchronized point cloud data and synchronized image information; Rasterize the synchronized point cloud data to obtain a first raster map, and rasterize the synchronized image information to obtain a second raster map; Fuse the first raster map and the second raster map according to a preset fusion method to obtain a fused raster map; Correct the point cloud data according to the fused raster map to obtain the target point cloud data.

3. The method according to claim 2, wherein Before rasterizing the synchronized point cloud data to obtain the first raster map, it further includes: Separate the point cloud data obtained by the radar sensor into dynamic point cloud data and static point cloud data; Correspondingly, the rasterizing the synchronized point cloud data to obtain the first raster map includes: Rasterize the detection area of the radar sensor, count the number of static points in each grid, and when the number of static points contained in the grid is greater than a first target threshold, the grid is an occupied grid, otherwise the grid is an invalid grid, and determine the first raster map through the occupied grid and the invalid grid.

4. The method according to claim 2, wherein The fusing the first raster map and the second raster map according to a preset fusion method to obtain a fused raster map includes: Calculate the target occupancy values of the corresponding grids in the fused raster map in sequence according to the weights of the first raster map and the second raster map in the fused raster map and the occupancy values of each grid in the first raster map and the second raster map; Divide the target occupancy value by the sum of the quantities of the first raster map and the second raster map to calculate the average occupancy values of the corresponding grids in the fused raster map in sequence. When the average occupancy value is greater than the second target threshold, mark the corresponding grid in the fused grid map as an occupied grid; otherwise, mark it as an invalid grid, thereby obtaining a fused grid map composed of the occupied grids and the invalid grids.

5. The method according to claim 3, characterized in that The step of correcting the point cloud data according to the fused grid map to obtain the target point cloud data includes: When the data point in the point cloud data corresponds to an occupied grid in the fused grid map and the current attribute of the data point is a moving point, correct the attribute of the data point to a static point; Use the point cloud data corresponding to the corrected static points and the static point cloud data together as the target point cloud data.

6. The method according to claim 3, characterized in that, The step of separating the point cloud data obtained by the radar sensor into dynamic point cloud data and static point cloud data includes: Obtain the measured Doppler velocity of each data point in the point cloud data; Calculate the target Doppler velocity of each data point according to the current vehicle speed; Calculate the difference between the measured Doppler velocity and the target Doppler velocity; When the absolute value of the difference is greater than the preset threshold, mark the data point as a moving point; when the absolute value of the difference is less than or equal to the preset threshold, mark the data point as a static point, and the point cloud data is divided into the dynamic point cloud data corresponding to the moving points and the static point cloud data corresponding to the static points.

7. The method according to claim 6, characterized in that, The step of calculating the target Doppler velocity of each data point according to the current vehicle speed includes: When the vehicle is in a straight driving state, the specific method for calculating the target Doppler velocity of each data point according to the current vehicle speed is: V di = V ego cosθ i Among them, V di is the target Doppler velocity; V ego is the vehicle driving speed, θ i is the sum of the azimuth angle and the installation angle of the i-th data point. The installation angle is the included angle between the connecting line of the origin of the vehicle coordinate system and the origin of the radar sensor or image sensor coordinate system and the Y-axis of the vehicle coordinate system. The azimuth angle is the angle passed by the connecting line of the i-th data point and the origin of the radar sensor or image sensor coordinate system when it rotates to the normal vector y-axis of the radar sensor or image sensor coordinate system along the minimum path. Among them, when rotating counterclockwise, the azimuth angle is positive, and when rotating clockwise, the azimuth angle is negative; When the vehicle is in a non - straight driving state, the specific method for calculating the target Doppler velocity of each data point according to the current vehicle speed is: V di = V y cosθ i + V x sinθ i Among them, V di is the target Doppler velocity, V x is the linear velocity of the vehicle driving speed in the X-axis direction of the vehicle coordinate system, V y is the linear velocity of the vehicle driving speed in the Y-axis direction of the vehicle coordinate system, θ i is the sum of the azimuth angle and the installation angle of the i-th data point.

8. The method according to any one of claims 1 to 7, characterized in that, The radar sensor is a millimeter - wave radar sensor and includes 1 forward radar sensor and 4 side - facing radar sensors, and the image sensor includes 1 camera.

9. An obstacle detection device applied to a vehicle, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the obstacle detection method according to any one of claims 1 to 8 above.

Images