Reversing obstacle recognition method, device and equipment and storage medium

Abstract

The invention discloses a car backing obstacle recognition method, device and equipment and a storage medium. Comprising the steps that radar point cloud data collected by a millimeter wave radar in the reversing process of a vehicle and a peripheral environment image shot by a camera of the vehicle are acquired; performing clustering and point cloud filling processing on the radar point cloud data based on a preset distance threshold between point clouds to obtain a clustering point cloud set; determining a point cloud screening range corresponding to the radar point cloud data according to parameters of a target frame detected by the radar point cloud data under a polar coordinate system and uncertainty corresponding to the target frame; screening target point cloud data in the point cloud screening range from a clustering point cloud set; and performing category identification and feature fusion on the target point cloud data and the peripheral environment image based on the RCS value of the target point cloud data to obtain an obstacle identification result. According to the invention, the accuracy of obstacle recognition can be enhanced, and the feature fusion capability of the point cloud is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of safe driving technology, and in particular to a method, device, equipment and storage medium for identifying a reversing obstacle. Background Art

[0002] In near-field perception such as automatic parking and vehicle sentry mode, multi-view cameras can provide detailed vehicle perception information under high-resolution conditions. However, it is difficult to achieve high-precision and robust three-dimensional object detection by relying solely on a single multi-view camera. For example, multi-view cameras are difficult to provide accurate depth information, and the image quality is greatly affected by weather, lighting, etc.

[0003] In order to improve the reversing safety and robustness in intelligent driving systems, multimodal sensors are usually used to obtain scene information for perception, such as surround view cameras, lidar, millimeter wave radar, etc. Millimeter wave radar is an economical and commonly used sensor that can provide relatively accurate depth information and speed information, and can provide high-quality millimeter wave point clouds under various weather and lighting conditions. Therefore, the millimeter wave radar-surround view camera multimodal joint perception solution has excellent perception capabilities and high cost performance, and is favored by many researchers and car manufacturers.

[0004] However, due to the huge differences between the 4D millimeter-wave radar and surround-view camera modalities, how to fuse the information of these two modalities to complete intelligent driving perception tasks (such as 3D object detection) with high precision and high robustness is a very big technical challenge. Summary of the invention

[0005] The embodiments of the present application provide a method, device, equipment and storage medium for identifying a reversing obstacle, which can be used to improve the density of point clouds, reduce errors caused by low angular resolution, enhance the accuracy of obstacle identification, and improve the feature fusion capability of point clouds. The technical solution is as follows:

[0006] On the one hand, an embodiment of the present application provides a method for identifying a reversing obstacle, the method comprising:

[0007] Acquire radar point cloud data collected by the millimeter-wave radar during the vehicle's reversing process, and surrounding environment images taken by the vehicle's camera;

[0008] Clustering and point cloud filling processing are performed on the radar point cloud data based on a preset distance threshold between point clouds to obtain a clustered point cloud set;

[0009] Determine a point cloud screening range corresponding to the radar point cloud data according to parameters of a target frame detected by the radar point cloud data in a polar coordinate system and an uncertainty corresponding to the target frame;

[0010] Filtering out target point cloud data within the point cloud screening range from the clustered point cloud set;

[0011] Based on the RCS value of the target point cloud data, category recognition and feature fusion are performed on the target point cloud data and the surrounding environment image to obtain an obstacle recognition result.

[0012] On the other hand, an embodiment of the present application provides a reversing obstacle recognition device, the device comprising:

[0013] A point cloud image acquisition module, used to acquire radar point cloud data collected by the millimeter wave radar during the vehicle's reversing process, and surrounding environment images taken by the vehicle's camera;

[0014] A point cloud set acquisition module, used for clustering and filling point cloud data based on a preset distance threshold between point clouds to obtain a clustered point cloud set;

[0015] A point cloud screening range determination module, used to determine a point cloud screening range corresponding to the radar point cloud data according to parameters of a target frame detected by the radar point cloud data in a polar coordinate system and an uncertainty corresponding to the target frame;

[0016] A target point cloud data screening module, used to screen out target point cloud data within the point cloud screening range from the clustered point cloud set;

[0017] The recognition result acquisition module is used to perform category recognition and feature fusion on the target point cloud data and the surrounding environment image based on the RCS value of the target point cloud data to obtain the recognition result of the obstacle.

[0018] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method described in the above aspects.

[0019] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the method described in the above aspects.

[0020] On the other hand, an embodiment of the present application provides a computer program product, which, when executed, is used to execute the method described in the above aspects.

[0021] In the technical solution provided in the embodiment of the present application, point cloud sparsity is solved by operations such as point cloud clustering and point cloud filling, and the error caused by low angular resolution is reduced. By using uncertainty to determine the point cloud screening range to screen the point cloud for subsequent feature fusion, the point cloud can be associated with the features under the polar coordinates of the image, which increases the richness of information and reduces the loss of point cloud information. Moreover, by introducing the RCS value of the radar, the accuracy of image classification and classification after image point cloud fusion is improved, thereby improving the feature fusion capability of the point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of a method for identifying a reversing obstacle provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of polar coordinate association provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of radar point feature distribution provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of the structure of a reversing obstacle recognition device provided in an embodiment of the present application;

[0026] Figure 5 A structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0028] Reference Figure 1 , shows a flowchart of the steps of a method for identifying a reversing obstacle provided by an embodiment of the present application. Figure 1 As shown, the reversing obstacle recognition method may include: step 101, step 102, step 103, step 104 and step 105.

[0029] Step 101: Acquire radar point cloud data collected by a millimeter-wave radar during a vehicle reversing process, and an image of the surrounding environment captured by a camera of the vehicle.

[0030] In this embodiment, millimeter wave radar (Millimeter Wave Radar) is a radio detection device operating in the millimeter wave band, and its frequency range is usually 30-300GHz, and the wavelength is between 1-10mm. This radar has the advantages of small size, low cost, all-weather operation, speed measurement capability and high distance resolution, and is widely used in the field of autonomous driving. In recent years, 4D millimeter wave radar technology has been developed. It adds the ability to measure the height of the target on the basis of traditional 3D radar, which can provide richer spatial information and help improve the perception ability of the autonomous driving system. 4D millimeter wave radar uses multiple-input multiple-output (MIMO) antenna technology and advanced signal processing algorithms to improve the radar's angular resolution and target detection accuracy.

[0031] During the vehicle's reversing process, the radar point cloud data collected by the vehicle's millimeter-wave radar and the surrounding environment images taken by the vehicle's camera can be obtained. Specifically, the millimeter-wave radar perceives the surrounding environment by emitting millimeter-wave signals and receiving their reflected signals. These reflected signals are converted into point cloud data, that is, a set of discrete points, each of which represents the position and attributes of a reflection source (such as a vehicle, pedestrian, road sign, etc.). These point cloud data provide the vehicle with rich environmental information, including the distance, speed, direction, etc. of the object. At the same time, as another important sensor, the camera also plays an important role in autonomous driving and assisted driving systems. The camera can capture detailed image information around the vehicle, including lane markings, roadside obstacles, pedestrians, traffic signals, etc.

[0032] In some examples, the millimeter wave radar can be activated and collect radar point cloud data when receiving a vehicle reversing instruction. For example, the instruction to trigger the millimeter wave radar to work can be pre-set in the vehicle system, that is, the millimeter wave radar is triggered to start working when the reversing is turned on to collect radar point cloud data during the vehicle reversing process. This triggering mechanism ensures that data is collected only when needed (such as when reversing), thereby saving computing resources and storage space.

[0033] In some examples, the vehicle's millimeter-wave radar can collect radar point cloud data in real time while the vehicle is driving, that is, when the vehicle is in driving state, the vehicle's millimeter-wave radar enters the working state to collect radar point cloud data around the vehicle in real time, etc. This method can continuously provide the vehicle with the surrounding environment.

[0034] It can be understood that the above examples are merely examples listed for a better understanding of the technical solutions of the embodiments of the present application, and are not intended to be the sole limitation to the embodiments.

[0035] After obtaining the radar point cloud data collected by the millimeter-wave radar during the vehicle's reversing process and the surrounding environment image taken by the vehicle's camera, step 102 is executed.

[0036] Step 102: clustering and point cloud filling processing are performed on the radar point cloud data based on a preset distance threshold between point clouds to obtain a clustered point cloud set.

[0037] After obtaining the radar point cloud data collected by the millimeter-wave radar during the vehicle's reversing process and the surrounding environment image taken by the vehicle's camera, the radar point cloud data can be clustered and filled based on the preset distance threshold between point clouds to obtain a clustered point cloud set. The implementation process can be described in detail in conjunction with the following specific implementation method.

[0038] In a specific implementation of the present application, the above step 102 may include:

[0039] Sub-step A1: Acquire all point cloud data collected by the millimeter-wave radar during the vehicle reversing process to form a point cloud cluster.

[0040] In this embodiment, all point cloud data collected by the millimeter wave radar of the vehicle during the vehicle reversing process can be obtained to form a point cloud cluster. Specifically, the point cloud can be stacked from the first frame to the current frame to form a point cloud cluster under the current frame.

[0041] Sub-step A2: Calculate the point cloud distance between any two point cloud data in the point cloud cluster.

[0042] After obtaining the point cloud cluster of the current frame, the point cloud distance between the point cloud data in the point cloud cluster can be calculated. In the specific implementation, a suitable distance measurement method can be selected to calculate the distance between the point cloud data, such as Euclidean distance, Manhattan distance, cosine similarity, etc. Then, the point cloud distance between the point cloud data in the point cloud cluster can be calculated according to the selected distance measurement method.

[0043] Sub-step A3: clustering the radar point cloud data according to the point cloud distance and the distance threshold to obtain clustered point cloud clusters.

[0044] After calculating the point cloud distances between two-by-two point cloud data in the obtained point cloud clusters, the radar point cloud data can be clustered according to the point cloud distances and a pre-set distance threshold to obtain clustered point cloud clusters. Specifically, the clustering process can be as follows: 1. Obtain the two points with the largest point cloud distances in the point cloud cluster. The two points with the largest distances are the first point and the second point. 2. Add the first point to the first point cloud cluster and add the second point to the second point cloud cluster. 3. Calculate the first distances and the second distances between the other points in the point cloud cluster except the first point and the second point and the first point and the second point respectively. 4. When the first distance is less than the second distance, add the other points to the first point cloud cluster. When the first distance is greater than or equal to the second distance, add the other points to the second point cloud cluster. 5. When the number of points in the first point cloud cluster and the second point cloud cluster reaches the threshold, or the distances between the remaining point cloud data in the point cloud cluster and the first point cloud and the second point are less than or equal to the distance threshold, the clustered first point cloud cluster and the second point cloud cluster are used as the clustered point cloud clusters. For example, calculate the distances between two-by-two samples (i.e., point clouds) in the same cluster (counted as c), and find the two samples a and b with the largest distances. Assign the samples a and b to different clusters c1 and c2 respectively, calculate the distances from the remaining other sample points in the original cluster (c) to a and b. If dis(a) (i.e., the distance corresponding to sample a) < dis(b) (i.e., the distance corresponding to sample b), then assign the sample points to c1, otherwise assign them to c2. Repeat the process of calculating distances and assigning sample points until the number of clusters is reached or less than or equal to the set sample distance threshold, and the clustered point cloud clusters c1 and c2 etc. can be obtained.

[0045] It can be understood that the above examples are only examples listed for better understanding the technical solutions of the embodiments of the present application and do not serve as the sole limitation of this embodiment.

[0046] Sub-step A4: Perform point cloud filling on the clustered point cloud clusters to obtain filled point cloud clusters, and use the filled point cloud clusters as the point cloud clustering set.

[0047] After obtaining the clustered point cloud clusters, point cloud filling can be performed on the clustered point cloud clusters to obtain filled point cloud clusters. Furthermore, the filled point cloud clusters can be used as the point cloud clustering set. Specifically, the two points with the farthest distances in the clustered point cloud clusters can be obtained. Randomly fill point clouds in a sphere with the line connecting the two points with the farthest distances as the diameter to obtain filled point cloud clusters, where the number of filled point clouds is N times the number of point cloud data in the clustered point cloud clusters, and N is a positive integer. For example, after obtaining the clustered point cloud cluster D, randomly fill point clouds in a sphere with the two points with the farthest distances as the diameter, and the number of filled points is 3 times the existing number in the existing point cloud cluster etc.

[0048] The embodiments of the present application solve the point cloud sparsity through operations such as hierarchical clustering and sphere filling, and reduce the errors caused by low angular resolution.

[0049] Step 103: Determine a point cloud screening range corresponding to the radar point cloud data according to the parameters of the target frame detected by the radar point cloud data in the polar coordinate system and the uncertainty corresponding to the target frame.

[0050] Uncertainty is a measure of the accuracy of a prediction or measurement and reflects the degree of uncertainty in the estimate of the true value of a quantity (such as angle or radial depth, etc.).

[0051] The polar coordinate system refers to a coordinate system consisting of poles, polar axes and polar diameters in a plane.

[0052] After obtaining the radar point cloud data, the point cloud screening range corresponding to the radar point cloud data can be determined according to the parameters of the target frame detected by the radar point cloud data in the polar coordinate system and the uncertainty corresponding to the target frame. The implementation process can be described in detail in conjunction with the following specific implementation method.

[0053] In a specific implementation of the present application, the above step 103 may include:

[0054] Sub-step B1: Obtain the predicted depth variance corresponding to the target box.

[0055] In this embodiment, the depth variance can be used to describe the degree of discreteness of the depth of the target frame in three-dimensional space (i.e., the vertical distance from the radar). It is an important indicator to measure the stability of the detected target frame in the depth direction. In this example, the depth variance refers to the degree of discreteness of the predicted depth values ​​at different time points, different perspectives, or different model predictions for the same "proposal" (i.e., the same obstacle target detection frame). Specifically, if the depth values ​​obtained by multiple predictions differ greatly, the depth variance will be higher, indicating that the predicted depth is unstable or there is a large uncertainty.

[0056] In practical applications, for radar point cloud data, the region of interest containing potential obstacles can be extracted according to task requirements to reduce the complexity of subsequent processing. Ground points are segmented from point cloud data to more accurately detect obstacles on the ground. This is usually achieved through plane fitting, RANSAC algorithm, etc. The remaining point cloud data is clustered to segment the point clouds belonging to the same obstacle. Commonly used clustering algorithms include Euclidean clustering, density clustering, etc. Finally, a three-dimensional bounding box (Bounding Box), i.e., a target box, can be fitted to each cluster (i.e., each obstacle) to represent the position, size, and shape of the obstacle.

[0057] The depth variance can be calculated as follows: for each point cloud in the target frame, calculate its depth value, that is, the vertical distance from the point to the radar. The depth values ​​of all point clouds in the target frame are averaged to obtain the depth mean of the target frame. According to the depth mean and the depth value of each point cloud, the depth variance of the target frame is calculated. The depth variance is the average of the squares of the differences between the depth values ​​of all point clouds in the target frame and the depth mean, which reflects the degree of discreteness of the target frame in the depth direction.

[0058] After obtaining the predicted depth variance corresponding to the target frame, sub-steps B2 and B3 are executed.

[0059] Sub-step B2: Obtain the polar coordinate angles corresponding to the leftmost coordinate point and the rightmost coordinate point of the target frame, respectively, wherein the leftmost coordinate point refers to the leftmost coordinate point on the target frame relative to the polar coordinate origin, and the rightmost coordinate point refers to the rightmost coordinate point on the target frame relative to the polar coordinate origin.

[0060] After obtaining the predicted depth variance corresponding to the target frame, the polar coordinate angles corresponding to the leftmost coordinate point and the rightmost coordinate point of the target frame can be obtained, where the leftmost coordinate point refers to the leftmost coordinate point on the target frame relative to the polar coordinate origin, and the rightmost coordinate point refers to the rightmost coordinate point on the target frame relative to the polar coordinate origin.

[0061] Sub-step B3: Obtain the first radial distance and the second radial distance of the target frame in the polar coordinate system, the first radial distance being the distance between the point of the target frame closest to the polar coordinate origin and the polar coordinate origin, and the second radial distance being the distance between the point of the target frame farthest from the polar coordinate origin and the polar coordinate origin.

[0062] At the same time, the first radial distance and the second radial distance of the target frame in the polar coordinate system can be obtained. The first radial distance is the distance between the point of the target frame closest to the polar coordinate origin and the polar coordinate origin, and the second radial distance is the distance between the point of the target frame farthest from the polar coordinate origin and the polar coordinate origin.

[0063] Sub-step B4: determining a point cloud screening range corresponding to the radar point cloud data according to the depth variance, the polar coordinate angle, the first radial distance, and the second radial distance.

[0064] Finally, the point cloud screening range corresponding to the radar point cloud data can be determined based on the depth variance, polar coordinate angle, first radial distance, and second radial distance. Specifically, the azimuth angle definition range corresponding to the radar point cloud data can be determined based on the polar coordinate angle and depth variance. And the radial depth definition range corresponding to the radar point cloud data can be determined based on the first radial distance, the second radial distance, and the depth variance. Finally, the point cloud screening range corresponding to the radar point cloud data is determined based on the azimuth angle definition range and the radial depth definition range. Figure 2 As shown, φ i Defines the range for the direction angle, r i is the radial depth definition range. The calculation formulas for the direction angle definition range and the radial depth definition range can be shown as the following formula (1) and formula (2) respectively:

[0065]

[0066] In the above formula (1), φ i Defines the range of direction angles, φ m,l is the polar coordinate angle corresponding to the leftmost coordinate point of the target box, φ m,r is the polar coordinate angle corresponding to the rightmost coordinate point of the target box, φ c is the angle between the center point of the target box and the origin of the polar coordinates, δ is the depth variance, γ is the preset minimum cloud point screening range, and σ is the preset hyperparameter for adjusting the radius size.

[0067]

[0068] In the above formula (2), r i Determines the radial depth range, r m,f is the first radial distance, r m,b is the second radial distance, r c is the distance between the center point of the target box and the origin of the polar coordinates, δ is the depth variance, γ is the preset minimum cloud point screening range, and σ is the preset hyperparameter for adjusting the radius size.

[0069] The embodiments of the present application can reduce the loss of point cloud information and increase the richness of point cloud information by increasing the uncertainty of the polar coordinate association between the radar point cloud and the image in the angular resolution direction.

[0070] After determining the point cloud screening range corresponding to the radar point cloud data according to the parameters of the target frame detected by the radar point cloud data in the polar coordinate system and the uncertainty corresponding to the target frame, step 104 is executed.

[0071] Step 104: Filter out target point cloud data within the point cloud screening range from the clustered point cloud set.

[0072] After determining the point cloud screening range corresponding to the radar point cloud data according to the parameters of the target frame detected by the radar point cloud data in the polar coordinate system and the uncertainty corresponding to the target frame, the target point cloud data within the point cloud screening range can be screened out from the clustered point cloud set for subsequent point cloud fusion processing.

[0073] After the target point cloud data within the point cloud screening range is screened out from the clustered point cloud set, step 105 is performed.

[0074] Step 105: Based on the RCS value of the target point cloud data, category recognition and feature fusion are performed on the target point cloud data and the surrounding environment image to obtain an obstacle recognition result.

[0075] The radar cross section (RCS) of millimeter-wave radar is a key parameter that describes the target's ability to reflect radar waves. RCS is a physical quantity that characterizes the intensity of the echo generated by the target when it is illuminated by radar waves. It is usually used to measure the "visibility" of the target. The size of RCS depends not only on the physical size of the target, but also on the shape of the target, material properties, frequency of the incident electromagnetic wave, and angle of incidence.

[0076] After the target point cloud data within the point cloud screening range is screened out from the clustered point cloud set, the target point cloud data and the surrounding environment image can be classified and fused based on the RCS value of the target point cloud data to obtain the obstacle recognition result.

[0077] In specific implementations, different objects have different scattering characteristics of radar signals due to their different shapes, materials and structures, so their RCS values ​​will also be different. In this example, the RCS values ​​corresponding to different obstacles can be pre-set, such as bicycles: 10dBsm, vehicles: 19-25dBsm, among which trucks: 15dBsm, cars: 10dBsm, motorcycles: 5dBsm, etc.

[0078] When obstacles are identified separately, they can be identified based on the acquired surrounding environment images. In this process, the RCS value can be combined to assist in determining the target classification, thereby improving the classification accuracy.

[0079] The classification and feature fusion process can be described in detail in combination with the following specific implementation methods.

[0080] In a specific implementation of the present application, the above step 105 may include:

[0081] Sub-step C1: Based on the RCS value of the target point cloud data and the surrounding environment image, the target frame is classified to obtain a classification result corresponding to the target frame.

[0082] In this embodiment, after the target point cloud data is obtained, the target frame can be classified according to the RCS value of the target point cloud data and the surrounding environment image of the vehicle to obtain a classification result corresponding to the target frame.

[0083] In practical applications, when using the model for classification and feature fusion, in addition to predicting the depth and speed of obstacles during the proposal and extraction of the final fused point cloud depth feature map, a regression detection head can be added to detect the RCS value to assist in determining target classification and improve classification accuracy. Specifically, for the model, the input format of the point cloud data can be defined as: (x, y, z, RCS, vx, vy), where (x, y, z) represents the three-dimensional coordinates of the input point cloud, vx is the velocity component of the point cloud in the x direction, and vy is the velocity component of the point cloud in the y direction. The added regression detection head can be used to detect the RCS value in the input point cloud data to assist in determining target classification.

[0084] Sub-step C2: performing feature fusion on the point cloud features of the target point cloud data and the image features of the surrounding environment image according to the classification result to obtain radar fusion features with classification semantic features.

[0085] Then, the point cloud features of the target point cloud data and the image features of the surrounding environment image can be fused according to the classification results to obtain radar fusion features with classification semantic features. Specifically, if there is a spatial or temporal correspondence between the target point cloud data and the surrounding environment image, feature matching and alignment are required to ensure the consistency of the fused features. Then, the classification results (or classification features) based on the target point cloud data can be fused with the classification results (or classification features) based on the surrounding environment image. On the basis of feature fusion, the semantic information in the classification results (such as category labels, probability distribution, etc.) is combined to generate radar fusion features with classification semantic features.

[0086] After the point cloud features of the target point cloud data and the image features of the surrounding environment image are fused according to the classification results to obtain radar fusion features with classification semantic features, sub-step C3 is executed.

[0087] Sub-step C3: Spread the radar fusion feature to multiple pixels on the BEV space image to obtain the recognition result of the obstacle.

[0088] BEV spatial image, or Bird's Eye View image, is a perspective image of an object or scene viewed from above. BEV spatial image is a method of converting objects and scenes in three-dimensional space into two-dimensional image representation. This representation method retains the relative position and size information of the object, but simplifies the three-dimensional space into a two-dimensional plane.

[0089] After the point cloud features of the target point cloud data and the image features of the surrounding environment image are fused according to the classification results to obtain the radar fusion features with classification semantic features, the radar fusion features can be spread to multiple pixels on the BEV space image to obtain the obstacle recognition results. The spreading process can be as follows Figure 3 As shown in FIG, when the millimeter wave data is enhanced using image data to generate radar features with semantic features, the voxel discretization operation guided by RCS spreads each radar point feature to multiple pixels on the BEV spatial image instead of a single pixel to increase the density of the BEV features. The specific operation is as follows: Figure 3 As shown. The scatter formula can be: Among them, (c x +c y ) is the coordinate of the radar point cloud data, V RCS is the RCS value corresponding to the radar, and r is the scatter diameter. When performing radar feature scattering, a patch can be defined around the projection position to enhance the representation capability of the millimeter wave point cloud using image features.

[0090] The embodiments of the present application make full use of the RCS measurement value based on the millimeter wave radar, thereby improving the classification accuracy and point cloud feature characterization capability, and being more robust.

[0091] The method for identifying obstacles when reversing provided in the embodiment of the present application obtains radar point cloud data collected by the millimeter-wave radar during the reversing process of the vehicle and the surrounding environment image taken by the camera of the vehicle. The radar point cloud data is clustered and point cloud filling is processed based on the preset distance threshold between the point clouds to obtain a clustered point cloud set. According to the uncertainty associated with the radar point cloud data and the target frame detected by the radar point cloud data in the polar coordinate system, the point cloud screening range corresponding to the radar point cloud data is determined. The target point cloud data within the point cloud screening range is screened out from the clustered point cloud set. The target point cloud data and the surrounding environment image are subjected to category recognition and feature fusion based on the RCS value of the target point cloud data to obtain the recognition result of the obstacle. The embodiment of the present application solves the sparsity of the point cloud through operations such as point cloud clustering and point cloud filling, and reduces the error caused by low angular resolution. By using uncertainty to determine the point cloud screening range to screen the point cloud for subsequent feature fusion, the point cloud can be associated with the features under the polar coordinates of the image, which increases the richness of information and reduces the loss of point cloud information. Moreover, by introducing the RCS value of the radar, the accuracy of image classification and image point cloud fusion is improved, and the feature fusion capability of the point cloud is enhanced.

[0092] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0093] Reference Figure 4 , shows a schematic diagram of the structure of a reversing obstacle recognition device provided by an embodiment of the present application. Figure 4 As shown, the reversing obstacle recognition device 400 may include the following modules:

[0094] The point cloud image acquisition module 410 is used to acquire radar point cloud data collected by the millimeter wave radar during the vehicle reversing process, and the surrounding environment image taken by the camera of the vehicle;

[0095] The point cloud set acquisition module 420 is used to perform clustering and point cloud filling processing on the radar point cloud data based on a preset distance threshold between point clouds to obtain a clustered point cloud set;

[0096] A point cloud screening range determination module 430 is used to determine a point cloud screening range corresponding to the radar point cloud data according to parameters of a target frame detected by the radar point cloud data in a polar coordinate system and an uncertainty corresponding to the target frame;

[0097] A target point cloud data screening module 440 is used to screen out target point cloud data within the point cloud screening range from the clustered point cloud set;

[0098] The recognition result acquisition module 450 is used to perform category recognition and feature fusion on the target point cloud data and the surrounding environment image based on the RCS value of the target point cloud data to obtain the recognition result of the obstacle.

[0099] Optionally, the point cloud set acquisition module includes:

[0100] A point cloud cluster acquisition unit, used to acquire all point cloud data collected by the millimeter wave radar during the vehicle reversing process to form a point cloud cluster;

[0101] A point cloud distance calculation unit, used for calculating the point cloud distance between any two point cloud data in the point cloud cluster;

[0102] A clustering point cloud cluster acquisition unit, used for clustering the radar point cloud data according to the point cloud distance and the distance threshold to obtain a clustering point cloud cluster;

[0103] The point cloud set acquisition unit is used to perform point cloud filling on the clustered point cloud clusters to obtain filled point cloud clusters, and use the filled point cloud clusters as the point cloud cluster set.

[0104] Optionally, the cluster point cloud cluster acquisition unit includes:

[0105] A maximum distance point acquisition subunit is used to acquire two points with the largest point cloud distance in the point cloud cluster, wherein the two points with the largest point distance are the first point and the second point;

[0106] a point adding subunit, used for adding the first point to a first point cloud cluster, and adding the second point to a second point cloud cluster;

[0107] a distance calculation subunit, configured to calculate a first distance and a second distance between other points in the point cloud cluster except the first point and the second point and the first point and the second point respectively;

[0108] a first point adding subunit, configured to add the other points to the first point cloud cluster when the first distance is smaller than the second distance;

[0109] a second point adding subunit, configured to add the other points to the second point cloud cluster when the first distance is greater than or equal to the second distance;

[0110] The clustering point cloud cluster acquisition subunit is used to use the clustered first point cloud cluster and the second point cloud cluster as the clustering point cloud cluster when the number of points in the first point cloud cluster and the second point cloud cluster reaches a threshold, or the distance between the remaining point cloud data in the point cloud cluster and the first point cloud and the second point cloud is less than or equal to the distance threshold.

[0111] Optionally, the point cloud set acquisition unit includes:

[0112] A farthest point acquisition subunit is used to acquire the two farthest points in the clustered point cloud cluster;

[0113] The filling point cloud cluster acquisition subunit is used to randomly fill point clouds in a sphere with a diameter of the line connecting the two points with the farthest distance from each other, to obtain the filled point cloud cluster, wherein the number of filled point clouds is N times the point cloud data in the clustered point cloud cluster, and N is a positive integer.

[0114] Optionally, the point cloud screening range determination module includes:

[0115] A depth variance acquisition unit, used to acquire the predicted depth variance corresponding to the target frame;

[0116] A polar coordinate angle acquisition unit, used to acquire polar coordinate angles corresponding to the leftmost coordinate point and the rightmost coordinate point of the target frame, respectively, wherein the leftmost coordinate point refers to the leftmost coordinate point on the target frame relative to the polar coordinate origin, and the rightmost coordinate point refers to the rightmost coordinate point on the target frame relative to the polar coordinate origin;

[0117] A radial distance acquisition unit, used to acquire a first radial distance and a second radial distance of the target frame in the polar coordinate system, wherein the first radial distance is the distance between the point of the target frame closest to the polar coordinate origin and the polar coordinate origin, and the second radial distance is the distance between the point of the target frame farthest from the polar coordinate origin and the polar coordinate origin;

[0118] A point cloud screening range determining unit is used to determine a point cloud screening range corresponding to the radar point cloud data according to the depth variance, the polar coordinate angle, the first radial distance and the second radial distance.

[0119] Optionally, the point cloud screening range determination unit includes:

[0120] A direction angle definition range determination subunit, used to determine the direction angle definition range corresponding to the radar point cloud data according to the polar coordinate angle and the depth variance;

[0121] a radial depth definition range determination subunit, configured to determine a radial depth definition range corresponding to the radar point cloud data according to the first radial distance, the second radial distance and the depth variance;

[0122] The point cloud screening range determination subunit is used to determine the point cloud screening range corresponding to the radar point cloud data according to the direction angle definition range and the radial depth definition range.

[0123] Optionally, the direction angle defining range determining subunit includes:

[0124] Based on the above formula (1), the direction angle definition range corresponding to the radar point cloud data is calculated as:

[0125]

[0126] In the above formula (1), φ i Defines the range of direction angles, φ m,l is the polar coordinate angle corresponding to the leftmost coordinate point of the target box, φ m,r is the polar coordinate angle corresponding to the rightmost coordinate point of the target box, φ c is the angle between the center point of the target box and the origin of the polar coordinates, δ is the depth variance, γ is the preset minimum cloud point screening range, and σ is the preset hyperparameter for adjusting the radius size.

[0127] Optionally, the radial depth limit range determination subunit includes:

[0128] Based on the above formula (2), the radial depth definition range corresponding to the radar point cloud data is calculated as:

[0129]

[0130] In the above formula (2), r i Determines the radial depth range, r m,f is the first radial distance, r m,b is the second radial distance, r c is the distance between the center point of the target box and the origin of the polar coordinates, δ is the depth variance, γ is the preset minimum cloud point screening range, and σ is the preset hyperparameter for adjusting the radius size.

[0131] Optionally, the recognition result acquisition module includes:

[0132] A classification result acquisition unit, configured to classify the target frame based on the RCS value of the target point cloud data and the surrounding environment image, and obtain a classification result corresponding to the target frame;

[0133] A fusion feature acquisition unit, used for performing feature fusion on the point cloud features of the target point cloud data and the image features of the surrounding environment image according to the classification result to obtain a radar fusion feature with classification semantic features;

[0134] The recognition result acquisition unit is used to spread the radar fusion feature to multiple pixels on the BEV space image to obtain the recognition result of the obstacle.

[0135] The reversing obstacle identification device provided in the embodiment of the present application obtains the radar point cloud data collected by the millimeter wave radar during the reversing process of the vehicle and the surrounding environment image taken by the camera of the vehicle. The radar point cloud data is clustered and the point cloud filling process is performed based on the preset distance threshold between the point clouds to obtain a clustered point cloud set. According to the uncertainty associated with the radar point cloud data and the target frame detected by the radar point cloud data in the polar coordinate system, the point cloud screening range corresponding to the radar point cloud data is determined. The target point cloud data within the point cloud screening range is screened out from the clustered point cloud set. The target point cloud data and the surrounding environment image are subjected to category recognition and feature fusion based on the RCS value of the target point cloud data to obtain the obstacle recognition result. The embodiment of the present application solves the sparsity of the point cloud through operations such as point cloud clustering and point cloud filling, and reduces the error caused by low angular resolution. By using uncertainty to determine the point cloud screening range to screen the point cloud for subsequent feature fusion, the point cloud can be associated with the features under the polar coordinates of the image, which increases the richness of information and reduces the loss of point cloud information. Moreover, by introducing the RCS value of the radar, the accuracy of image classification and image point cloud fusion is improved, and the feature fusion capability of the point cloud is enhanced.

[0136] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0137] Please refer to Figure 5 , which shows a block diagram of a computer device provided in one embodiment of the present application. The computer device can be used to implement the backing obstacle identification method provided in the above embodiment. The computer device can be a PC or a server, or other devices with data processing and storage capabilities. Specifically:

[0138] The computer device 500 includes a central processing unit (CPU) 501, a system memory 504 including a random access memory (RAM) 502 and a read-only memory (ROM) 503, and a system bus 505 connecting the system memory 504 and the central processing unit 501. The computer device 500 also includes a basic input / output system (I / O system) 506 that helps transfer information between various components in the computer, and a large-capacity storage device 507 for storing an operating system 513, application programs 514, and other program modules 515.

[0139] The basic input / output system 506 includes a display 508 for displaying information and an input device 509 such as a mouse and a keyboard for user inputting information. The display 508 and the input device 509 are connected to the central processing unit 501 through an input / output controller 510 connected to the system bus 505. The basic input / output system 506 may also include an input / output controller 510 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 510 also provides output to a display screen, a printer, or other types of output devices.

[0140] The mass storage device 507 is connected to the central processing unit 501 through a mass storage controller (not shown) connected to the system bus 505. The mass storage device 507 and its associated computer readable media provide non-volatile storage for the computer device 500. That is, the mass storage device 507 may include a computer readable medium (not shown) such as a hard disk or a CD-ROM drive.

[0141] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, DVD or other optical storage, tape cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media is not limited to the above. The above-mentioned system memory 504 and mass storage device 507 can be collectively referred to as memory.

[0142] According to various embodiments of the present application, the computer device 500 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 500 can be connected to the network 512 through the network interface unit 511 connected to the system bus 505, or the network interface unit 511 can be used to connect to other types of networks or remote computer systems (not shown).

[0143] The memory further includes one or more programs, which are stored in the memory and configured to be executed by one or more processors. The one or more programs include instructions for executing the reverse obstacle recognition method.

[0144] In an example embodiment, a computer device is also provided, the computer device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set. The at least one instruction, at least one program, code set or instruction set is configured to be executed by one or more processors to implement the above-mentioned reversing obstacle recognition method.

[0145] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. When the at least one instruction, the at least one program, the code set or the instruction set is executed by a processor of a computer device, the above-mentioned reversing obstacle recognition method is implemented.

[0146] Optionally, the computer-readable storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0147] In an exemplary embodiment, a computer program product is also provided. When the computer program product is executed, it is used to implement the above-mentioned reversing obstacle recognition method.

[0148] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0149] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for identifying a reversing obstacle, characterized in that: The method comprises: Acquire radar point cloud data collected by the millimeter-wave radar during the vehicle's reversing process, and surrounding environment images taken by the vehicle's camera; Clustering and point cloud filling processing are performed on the radar point cloud data based on a preset distance threshold between point clouds to obtain a clustered point cloud set; Determine a point cloud screening range corresponding to the radar point cloud data according to parameters of a target frame detected by the radar point cloud data in a polar coordinate system and an uncertainty corresponding to the target frame; Filtering out target point cloud data within the point cloud screening range from the clustered point cloud set; Based on the RCS value of the target point cloud data, category recognition and feature fusion are performed on the target point cloud data and the surrounding environment image to obtain an obstacle recognition result.

2. The method according to claim 1, characterized in that The step of clustering and filling the radar point cloud data based on a preset distance threshold between point clouds to obtain a clustered point cloud set includes: Acquire all point cloud data collected by the millimeter-wave radar during the vehicle reversing process to form a point cloud cluster; Calculating the point cloud distance between any two point cloud data in the point cloud cluster; Clustering the radar point cloud data according to the point cloud distance and the distance threshold to obtain clustered point cloud clusters; Point cloud filling is performed on the clustered point cloud cluster to obtain a filled point cloud cluster, and the filled point cloud cluster is used as the point cloud cluster set.

3. The method according to claim 2, characterized in that The step of clustering the radar point cloud data according to the point cloud distance and the distance threshold to obtain a clustered point cloud cluster includes: Obtain two points in the point cloud cluster with the largest point cloud distance, wherein the two points with the largest point cloud distance are the first point and the second point; Adding the first point to a first point cloud cluster, and adding the second point to a second point cloud cluster; Calculating first distances and second distances between other points in the point cloud cluster except the first point and the second point and the first point and the second point respectively; When the first distance is smaller than the second distance, the other points are added to the first point cloud cluster; When the first distance is greater than or equal to the second distance, adding the other points to the second point cloud cluster; When the number of points in the first point cloud cluster and the second point cloud cluster reaches a threshold, or the distance between the remaining point cloud data in the point cloud cluster and the first point cloud and the second point cloud is less than or equal to the distance threshold, the clustered first point cloud cluster and the second point cloud cluster are used as the clustered point cloud cluster.

4. The method according to claim 2, characterized in that: The step of performing point cloud filling on the clustered point cloud cluster to obtain a filled point cloud cluster includes: Obtain the two points with the farthest distance in the cluster point cloud cluster; Point clouds are randomly filled in a sphere whose diameter is a line connecting the two points with the greatest distance, to obtain the filled point cloud cluster, wherein the number of filled point clouds is N times the number of point cloud data in the clustered point cloud cluster, and N is a positive integer.

5. The method according to claim 1, characterized in that Determining a point cloud screening range corresponding to the radar point cloud data according to parameters of a target frame detected by the radar point cloud data in a polar coordinate system and an uncertainty corresponding to the target frame includes: Obtaining the predicted depth variance corresponding to the target frame; Obtain the polar coordinate angles corresponding to the leftmost coordinate point and the rightmost coordinate point of the target frame, respectively, wherein the leftmost coordinate point refers to the leftmost coordinate point on the target frame relative to the polar coordinate origin, and the rightmost coordinate point refers to the rightmost coordinate point on the target frame relative to the polar coordinate origin; Obtain a first radial distance and a second radial distance of the target frame in the polar coordinate system, wherein the first radial distance is the distance between the point of the target frame closest to the polar coordinate origin and the polar coordinate origin, and the second radial distance is the distance between the point of the target frame farthest from the polar coordinate origin and the polar coordinate origin; A point cloud screening range corresponding to the radar point cloud data is determined according to the depth variance, the polar coordinate angle, the first radial distance, and the second radial distance.

6. The method according to claim 5, characterized in that The step of determining a point cloud screening range corresponding to the radar point cloud data according to the depth variance, the polar coordinate angle, the first radial distance, and the second radial distance includes: Determine a direction angle boundary range corresponding to the radar point cloud data according to the polar coordinate angle and the depth variance; Determine a radial depth definition range corresponding to the radar point cloud data according to the first radial distance, the second radial distance and the depth variance; A point cloud screening range corresponding to the radar point cloud data is determined according to the direction angle definition range and the radial depth definition range.

7. The method according to claim 6, characterized in that The determining, according to the polar coordinate angle and the depth variance, a direction angle definition range corresponding to the radar point cloud data comprises: Based on the above formula (1), the direction angle definition range corresponding to the radar point cloud data is calculated as: In the above formula (1), φ i Defines the range of direction angles, φ m,l is the polar coordinate angle corresponding to the leftmost coordinate point of the target box, φ m,r is the polar coordinate angle corresponding to the rightmost coordinate point of the target box, φ c is the angle between the center point of the target box and the origin of the polar coordinates, δ is the depth variance, γ is the preset minimum cloud point screening range, and σ is the preset hyperparameter for adjusting the radius size.

8. The method according to claim 6, characterized in that The determining, according to the first radial distance, the second radial distance and the depth variance, a radial depth definition range corresponding to the radar point cloud data includes: Based on the above formula (2), the radial depth definition range corresponding to the radar point cloud data is calculated as: In the above formula (2), r i The radial depth range is defined as r m,f is the first radial distance, r m,b is the second radial distance, r c is the distance between the center point of the target box and the origin of the polar coordinates, δ is the depth variance, γ is the preset minimum cloud point screening range, and σ is the preset hyperparameter for adjusting the radius size.

9. The method according to any one of claims 1 to 8, characterized in that: The performing category recognition and feature fusion on the target point cloud data and the surrounding environment image based on the RCS value of the target point cloud data to obtain an obstacle recognition result includes: Based on the RCS value of the target point cloud data and the surrounding environment image, the target frame is classified to obtain a classification result corresponding to the target frame; Performing feature fusion on the point cloud features of the target point cloud data and the image features of the surrounding environment image according to the classification result to obtain radar fusion features with classification semantic features; The radar fusion feature is spread to multiple pixels on the BEV space image to obtain the recognition result of the obstacle.

10. A reversing obstacle recognition device, characterized in that: The device comprises: A point cloud image acquisition module, used to acquire radar point cloud data collected by the millimeter wave radar during the vehicle's reversing process, and surrounding environment images taken by the vehicle's camera; A point cloud set acquisition module, used for clustering and filling point cloud data based on a preset distance threshold between point clouds to obtain a clustered point cloud set; A point cloud screening range determination module, used to determine a point cloud screening range corresponding to the radar point cloud data according to parameters of a target frame detected by the radar point cloud data in a polar coordinate system and an uncertainty corresponding to the target frame; A target point cloud data screening module, used to screen out target point cloud data within the point cloud screening range from the clustered point cloud set; The recognition result acquisition module is used to perform category recognition and feature fusion on the target point cloud data and the surrounding environment image based on the RCS value of the target point cloud data to obtain the recognition result of the obstacle.

11. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 9.

Images