Point cloud target processing method, terminal equipment and storage medium

By projecting the target contour points in the point cloud frame to the image frame and adjusting the number of projected contour points, the problem of dynamic obstacle size information jumping is solved, and the target detection accuracy and operation stability of intelligent driving cars are improved.

CN119027437APending Publication Date: 2024-11-26ZHEJIANG LEAPMOTOR TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410906718.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-05
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In intelligent driving cars, the point cloud data size information of dynamic obstacles continues to jump, affecting the stability of the subsequent operations of the car. It is difficult for the existing technology to effectively solve this problem.

Method used

By projecting the target contour points in the point cloud frame into the image frame, the number of projected contour points is adjusted using the image detection results, the target contour points in the point cloud frame are optimized, and the accuracy of target detection is improved.

Benefits of technology

It improves the accuracy of intelligent driving cars for dynamic obstacle size information, enhances the stability of subsequent operations, and reduces resource consumption.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027437A_ABST
    Figure CN119027437A_ABST
Patent Text Reader

Abstract

The invention discloses a point cloud target processing method, terminal equipment and a storage medium, and the method comprises the steps: determining a point cloud detection result and an image detection result which belong to the same target from a point cloud frame and an image frame at the same moment, and enabling the point cloud detection result to comprise a first target contour point corresponding to the target; the image detection result comprises a target detection frame corresponding to the target; projecting the first target contour point into the image frame to obtain a projection contour point; according to the position relation between the projection contour points and the target detection frame, the number of the projection contour points is adjusted, and final projection contour points are obtained; and determining a corresponding second target contour point from the point cloud frame by using the final projection contour point. According to the scheme, the accuracy and the stability of the obtained contour point corresponding to the target are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of point cloud data processing, and in particular to a point cloud target processing method, terminal device and storage medium. Background Art

[0002] When performing assisted driving, the car needs to sense the surrounding obstacles. One way of perception is to obtain the point cloud data collected by the radar sensor, then determine the size and position of the obstacle from the point cloud data, and finally plan a safe driving path based on the detection results.

[0003] If the obstacle is a dynamic obstacle, the size of the obstacle obtained according to the detection results may be constantly changing, seriously affecting the stability of subsequent operations of the intelligent driving vehicle. Summary of the invention

[0004] The present application provides a point cloud target processing method, terminal device and storage medium.

[0005] A technical solution adopted in this application is to provide a method for processing a point cloud target, the method comprising:

[0006] Determine a point cloud detection result and an image detection result belonging to the same target from the point cloud frame and the image frame at the same time, wherein the point cloud detection result includes a first target contour point corresponding to the target, and the image detection result includes a target detection frame corresponding to the target;

[0007] Projecting the first target contour point into the image frame to obtain a projected contour point;

[0008] According to the positional relationship between the projection contour points and the target detection frame, the number of the projection contour points is adjusted to obtain final projection contour points;

[0009] The final projected contour points are used to determine corresponding second target contour points from the point cloud frame.

[0010] Optionally, adjusting the number of the projection contour points according to the positional relationship between the projection contour points and the target detection frame to obtain the final projection contour points includes:

[0011] Acquire a first distribution mean of the projection contour points and / or a first distribution variance of the projection contour points; and a second distribution mean and / or a second distribution variance of the target detection frame;

[0012] In response to a difference between the first distribution mean and the second distribution mean being greater than or equal to a first preset difference, adjusting the number of the projection contour points to obtain the final projection contour points;

[0013] and / or,

[0014] In response to the difference between the first distribution variance and the second distribution variance being greater than or equal to a second preset difference, the number of the projection contour points is adjusted to obtain the final projection contour points.

[0015] Optionally, in response to the absolute value of the difference between the first distribution mean and the second distribution mean being greater than or equal to a first preset difference, adjusting the number of the projection contour points to obtain the final projection contour points includes:

[0016] Determining a first missing direction and a first adjustment step size of the projected contour point relative to the target detection frame based on a difference between the first distribution mean and the second distribution mean;

[0017] In the point cloud frame, according to the first target contour point, a first contour point sampling area is determined along the first missing direction, wherein a size of the first contour point sampling area is determined by the first adjustment step size;

[0018] Sampling data points in the first contour point sampling area and projecting the data points to the image frame to obtain first projected sampling points;

[0019] The number of the projection contour points is adjusted using the first projection sampling points to obtain the final projection contour points.

[0020] Optionally, in response to the absolute value of the difference between the first distribution variance and the second distribution variance being greater than or equal to a second preset difference, adjusting the number of the projection contour points to obtain the final projection contour points includes:

[0021] Determining a second missing direction and a second adjustment step size based on a difference between the first distribution variance and the second distribution variance;

[0022] The second missing direction and the second adjustment step length are used to adjust the number of the projection contour points to obtain the final projection contour points.

[0023] Optionally, the second missing direction is a missing direction of the projected contour point relative to the target detection frame;

[0024] The adjusting the number of the projection contour points by using the second missing direction and the second adjustment step to obtain the final projection contour points includes:

[0025] In the point cloud frame, according to the first target contour point, a second contour point sampling area is determined along the second missing direction, wherein the size of the second contour point sampling area is determined by the second adjustment step size;

[0026] Sampling data points in the second contour point sampling area and projecting them into the image frame to obtain second projected sampling points;

[0027] The second projection sampling points are used to adjust the number of the projection contour points to obtain the final projection contour points.

[0028] Optionally, the determining a second missing direction and a second adjustment step size based on a difference between the first distribution variance and the second distribution variance includes:

[0029] In response to the first distribution variance being greater than the second distribution variance, and the difference between the first distribution mean and the second distribution mean being less than the first preset difference, determining a second missing direction and a second adjustment step length of the target detection frame relative to the projected contour point;

[0030] The adjusting the number of the projection contour points by using the second missing direction and the second adjustment step to obtain the final projection contour points includes:

[0031] Along the opposite direction of the second missing direction, using the second adjustment step size, the projected contour points located outside the preset distance from the center point of the target detection frame are deleted to obtain remaining projected contour points;

[0032] In the point cloud frame, along the second missing direction, taking the distance between the boundary of the target detection frame and the preset distance as a reference, determining a second contour point sampling area;

[0033] Sampling data points in the second contour point sampling area and projecting them into the image frame to obtain second projected sampling points;

[0034] The second projection sampling points are added to the remaining projection contour points to obtain the final projection contour points.

[0035] Optionally, the first distribution mean includes a first sub-distribution mean of the projected contour points in the horizontal direction, and / or a second sub-distribution mean of the projected contour points in the vertical direction;

[0036] The second distribution mean includes a third sub-distribution mean of the target detection frame in the horizontal direction, and / or a fourth sub-distribution mean of the target detection frame in the vertical direction;

[0037] The first distribution variance includes a first sub-distribution variance of the projected contour points in a horizontal direction, and / or a second sub-distribution variance of the projected contour points in a vertical direction;

[0038] The second distribution variance includes a third sub-distribution variance of the target detection frame in a horizontal direction, and / or a fourth sub-distribution variance of the target detection frame in a vertical direction.

[0039] Optionally, projecting the first target contour point into the image frame to obtain the projected contour point includes:

[0040] Acquire a projection matrix between the point cloud frame and the image frame;

[0041] Based on the projection matrix, the first target contour points are projected from the point cloud frame to the image frame to obtain projected contour points.

[0042] Another technical solution adopted by the present application is to provide a terminal device, the terminal device comprising a memory and a processor connected to the memory;

[0043] The memory is used to store program data, and the processor is used to execute the program data to implement the point cloud target processing method as described above.

[0044] Another technical solution adopted by the present application is to provide a computer storage medium, wherein the computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the processing method of the point cloud target as described above.

[0045] The beneficial effects of the present application are: the first target contour point corresponding to the target in the point cloud frame is projected into the image frame at the same time to obtain the projection contour point, and the number of projection contour points is adjusted by comparing the positional relationship between the target detection frame corresponding to the target in the image frame and the projection contour point to obtain the final projection contour point, and then the second target contour point corresponding to the final projected contour point is determined from the point cloud frame. The point cloud target processing method provided by the present application optimizes the contour points of the target in the point cloud frame using the image detection results, so that the final second target contour point is more accurate, thereby improving the stability of subsequent operations of the intelligent driving car. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 It is a flowchart of an embodiment of a method for processing a point cloud target provided by the present application;

[0048] Figure 2It is a flowchart of another embodiment of the method for processing point cloud targets provided by the present application;

[0049] Figure 3 It is a schematic diagram of the distribution of projection points and image detection boxes;

[0050] Figure 4 It is a structural diagram of an embodiment of a terminal device provided by the present application;

[0051] Figure 5 It is a structural diagram of an embodiment of a computer storage medium provided by the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] The terms "first", "second", etc. in this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods or devices.

[0054] In addition, although the terms "first", "second", etc. are used many times in this application to describe various data (or various applications or various instructions or various operations), etc., these data (or applications or instructions or operations) should not be limited by these terms. These terms are only used to distinguish one type of data (or application or instruction or operation) from another type of data (or application or instruction or operation).

[0055] In intelligent driving scenarios, radar sensors such as millimeter-wave radars or lidars have relatively accurate ranging capabilities. For non-360-degree radar sensors, there will inevitably be a field of view (FOV) boundary. If the target is located at the boundary of the FOV, the point cloud corresponding to the target obtained by the radar sensor may only be a part of the target. When the target passes through the FOV of the radar sensor, the size information output by the radar sensor will continue to jump, affecting subsequent decision-making operations.

[0056] In the related art, in order to improve the accuracy of the target size information output by the radar sensor, the size information of the same target in several consecutive point cloud frames is obtained, and the size information is used to adjust the size of the same target in each point cloud frame. After a period of work and research, the inventor of the present application found that this method can obtain more accurate size information for static point cloud targets, because the position and size of static point cloud targets will not change. For dynamic point cloud targets, the size and position output by the radar sensor change at different times, making it difficult for this method to use the constantly changing size data to obtain the accurate size of the dynamic point cloud target. In addition, in this method, the position of the radar sensor is constantly changing. In order to obtain information about obstacles, the point cloud frames need to be continuously synchronized in time and space, and this operation will result in a large consumption of resources.

[0057] Please refer to Figure 1 , Figure 1 It is a flow chart of an embodiment of a method for processing point cloud targets provided in the present application.

[0058] like Figure 1 As shown, the method for processing the point cloud target in the embodiment of the present application may specifically include the following steps:

[0059] S1, determining the point cloud detection results and image detection results belonging to the same target from the point cloud frame and image frame at the same time.

[0060] The point cloud detection result includes the first target contour point corresponding to the target, and the image detection result includes the target detection frame corresponding to the target. The target corresponds to the obstacle mentioned above.

[0061] The point cloud target processing method provided in the present application is mainly performed by a point cloud target processing device. In some embodiments, the point cloud target processing device may be the radar sensor itself. In some embodiments, the point cloud target processing device may be a device that is communicatively connected to the radar sensor. For example, the device may be a device for monitoring images, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, and an autonomous vehicle, a robot, a security system, or any one or more of glasses and helmets for augmented reality or virtual reality. In some possible implementations, the point cloud target processing method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0062] It can be understood that the point cloud target processing device is connected to the camera for communication.

[0063] Optionally, the radar sensor may be any one or more of a laser radar and a millimeter wave radar.

[0064] Specifically, the point cloud target processing device obtains the point cloud frame collected by the radar sensor and the image frame collected by the camera at the same time. The point cloud target processing device performs target detection on the image frame to obtain the image detection result, and performs target detection or target segmentation on the point cloud frame to obtain the point cloud detection result.

[0065] In some embodiments, the point cloud target processing device inputs the image frame into the image target detection network for target detection to obtain a target detection result. Exemplarily, the image target detection network can be a YOLO-based target detection network, or other common image target detection networks, which are not limited here.

[0066] In some embodiments, the point cloud target processing device inputs the point cloud frame into a point cloud target detection network or a point cloud segmentation network to obtain a point cloud detection result. Exemplarily, the point cloud target detection network or the point cloud segmentation network may be PointPillar, pointNet or a variant thereof, or other common point cloud target detection networks or point cloud segmentation networks, which are not limited here.

[0067] In some possible embodiments, the point cloud detection result also includes a 3D object detection box.

[0068] It should be noted that the first target contour points are some data points in the point cloud corresponding to the target, which are used to characterize the contour information of the target (such as the size of the target), similar to the edge of the target in a 2D image.

[0069] In some possible embodiments, the first target contour points are obtained by a point cloud target processing device performing a sampling operation on data points corresponding to the target.

[0070] In some embodiments, the point cloud object processing device determines whether the first object and the second object are the same object by obtaining a first position of the first object in the image frame and a second position of the second object in the point cloud frame, and further determining the similarity between the first position and the second position. Exemplarily, in response to the similarity between the first position and the second position being greater than or equal to a similarity threshold, the point cloud object processing device determines that the first object and the second object are the same object.

[0071] S2, projecting the first target contour point into the image frame to obtain a projected contour point.

[0072] The methods for fusing point cloud data with image data are mainly divided into two categories. One of the methods is to project image data onto point cloud data, such as obtaining three-dimensional information from a 2D detection frame of image data, such as performing vision-based depth estimation. However, the depth estimation results of this method have poor accuracy.

[0073] Another method is to project the point cloud data onto the image data. This embodiment mainly utilizes the latter method to obtain the mapping relationship between the point cloud frame and the image frame. Using the mapping relationship, the first target contour point of 3D is projected onto the 2D projection frame to obtain the projection contour point. In addition, the depth information of the projection contour point is also included after the projection to facilitate possible subsequent resampling operations.

[0074] S3, adjusting the number of projection contour points according to the positional relationship between the projection contour points and the target detection frame to obtain the final projection contour points.

[0075] Specifically, the point cloud target processing device increases or decreases the number of projection contour points according to the positional relationship between the projection contour points and the target detection frame, so that the overall position of the adjusted projection contour points is close to the target detection frame, and / or the size of the adjusted projection contour points is close to the size of the target detection frame. Furthermore, the point cloud target processing device uses the adjusted projection contour points as the final projection contour points.

[0076] S4, using the final projected contour point to determine the corresponding second target contour point from the point cloud frame.

[0077] Specifically, the point cloud target processing device uses the mapping relationship between the point cloud frame and the image frame to determine the data point corresponding to the final projection contour point in the point cloud frame, that is, the second target contour point.

[0078] In some possible embodiments, the point cloud object processing device obtains the size of the second target contour point and outputs the size of the second target contour point to a downstream device. For example, the downstream device may be a vehicle-mounted terminal, and the downstream device may obtain the size of the second target contour point.

[0079] In the above scheme, the first target contour point corresponding to the target in the point cloud frame is projected into the image frame at the same time to obtain the projection contour point, and the number of projection contour points is adjusted by comparing the positional relationship between the target detection frame corresponding to the target in the image frame and the projection contour point to obtain the final projection contour point, and then the second target contour point corresponding to the final projected contour point is determined from the point cloud frame. The point cloud target processing method provided in the present application optimizes the contour points of the target in the point cloud frame using the image detection results, so that the final second target contour point is more accurate, thereby improving the stability of subsequent operations of the intelligent driving car.

[0080] Another embodiment of the method for processing a point cloud target provided by the present application may specifically include the following steps:

[0081] S11, determining the point cloud detection result and the image detection result belonging to the same target from the point cloud frame and the image frame at the same time.

[0082] S12, projecting the first target contour point into the image frame to obtain a projected contour point.

[0083] S13, obtaining a first distribution mean of the projection contour points.

[0084] This embodiment is based on the assumption that the projected contour points obey the Gaussian distribution, that is, the distribution of the projected contour points satisfies N′, where μ′ is the first distribution mean.

[0085] Optionally, the first distribution mean includes a first sub-distribution mean of the projected contour points in the horizontal direction, and / or a second sub-distribution mean of the projected contour points in the vertical direction.

[0086] In some embodiments, the first distribution mean satisfies the following relationship:

[0087]

[0088] Among them, μ x ′ is the mean of the first subdistribution, μ y ′ is the mean of the second sub-distribution.

[0089] S14, obtaining a second distribution mean of the target detection box.

[0090] This embodiment is based on the assumption that the target detection box also obeys the Gaussian distribution, that is, the distribution of the target detection box satisfies N(μ,σ 2 ), where μ is the second distribution mean. In this embodiment, the second distribution mean can be expressed as the coordinates of the center point of the target detection frame.

[0091] Optionally, the second distribution mean includes a third sub-distribution mean of the target detection box in the horizontal direction, and / or a fourth sub-distribution mean of the target detection box in the vertical direction.

[0092] In some embodiments, the second distribution mean satisfies the following relationship:

[0093]

[0094] Among them, μ x is the mean of the third sub-distribution, μ y In this embodiment, the third sub-distribution mean can be expressed as the horizontal coordinate of the center point of the target detection frame, and the fourth sub-distribution mean can be expressed as the vertical coordinate of the center point of the target detection frame.

[0095] S15 , in response to the difference between the first distribution mean and the second distribution mean being greater than or equal to the first preset difference, adjusting the number of projection contour points to obtain final projection contour points.

[0096] Specifically, in response to the difference between the first distribution mean and the second distribution mean being greater than or equal to the first preset difference, the point cloud object processing device increases or decreases the number of projection contour points to obtain final projection contour points.

[0097] Among them, the difference between the third distribution mean of the final projection contour point and the second distribution mean is smaller than the first preset difference, that is, the center point of the final projection contour point is close to the center point of the target detection frame.

[0098] In some embodiments, the first preset difference includes a first sub-preset difference and a second sub-preset difference.

[0099] In some embodiments, in response to the difference between the first sub-distribution mean and the third sub-distribution mean being greater than the first sub-preset difference, the point cloud object processing device increases or decreases the number of projection contour points to obtain the final projection contour points. It can be understood that the horizontal coordinate of the center point of the final projection contour point is close to the horizontal coordinate of the center point of the target detection frame.

[0100] In some embodiments, in response to the difference between the second sub-distribution mean and the fourth sub-distribution mean being greater than the second sub-preset difference, the point cloud object processing device increases or decreases the number of projection contour points to obtain the final projection contour points. It can be understood that the ordinate of the center point of the final projection contour point is close to the ordinate of the center point of the target detection frame.

[0101] S16, using the final projected contour point to determine the corresponding second target contour point from the point cloud frame.

[0102] In the above scheme, the first target contour point corresponding to the target in the point cloud frame is projected into the image frame at the same time to obtain the projection contour point, and the number of projection contour points is adjusted by comparing the positional relationship between the target detection frame corresponding to the target in the image frame and the projection contour point to obtain the final projection contour point, and then the second target contour point corresponding to the final projected contour point is determined from the point cloud frame. The point cloud target processing method provided in the present application optimizes the contour points of the target in the point cloud frame using the image detection results, so that the final second target contour point is more accurate, thereby improving the stability of subsequent operations of the intelligent driving car.

[0103] Furthermore, by adjusting the number of projection contour points, the distribution mean of the final projection contour points is made close to the mean of the target detection frame, thereby improving the accuracy of the overall position of the projection contour points.

[0104] Another embodiment of the method for processing a point cloud target provided by the present application may specifically include the following steps:

[0105] S21, determining the point cloud detection results and image detection results belonging to the same target from the point cloud frame and the image frame at the same time.

[0106] S22, projecting the first target contour point into the image frame to obtain a projected contour point.

[0107] S23, obtaining a first distribution variance of the projection contour points.

[0108] This embodiment is based on the assumption that the projection contour points obey the Gaussian distribution, that is, the distribution of the projection contour points satisfies N′, where (σ||′) 2 is the variance of the first distribution.

[0109] Optionally, the first distribution variance includes a first sub-distribution variance of the projected contour points in a horizontal direction, and / or a second sub-distribution variance of the projected contour points in a vertical direction.

[0110] In some embodiments, the first distribution variance satisfies the following relationship:

[0111]

[0112] Among them, (σ′ x ) 2 is the variance of the first subdistribution, (σ′ y ) 2 is the variance of the second sub-distribution.

[0113] S24, obtaining a second distribution variance of the target detection box.

[0114] This embodiment is based on the assumption that the target detection box also obeys the Gaussian distribution, that is, the distribution of the target detection box satisfies N(μ,σ 2 ), where σ 2 is the second distribution variance. In this embodiment, the second distribution variance can be expressed as the size of the target detection frame (the width and / or height of the target detection frame).

[0115] Optionally, the second distribution variance includes a third sub-distribution variance of the target detection box in the horizontal direction, and / or a fourth sub-distribution variance of the target detection box in the vertical direction.

[0116] In some embodiments, the second distribution variance satisfies the following relationship:

[0117]

[0118] in, is the variance of the third sub-distribution, μ y In this embodiment, the third sub-distribution variance can be expressed as the width of the target detection frame, and the fourth sub-distribution variance can be expressed as the height of the target detection frame.

[0119] S25 , in response to the difference between the first distribution variance and the second distribution variance being greater than or equal to the second preset difference, adjusting the number of projection contour points to obtain final projection contour points.

[0120] Specifically, in response to the difference between the first distribution variance and the second distribution variance being greater than or equal to the second preset difference, the point cloud object processing device increases or decreases the number of projection contour points to obtain final projection contour points.

[0121] The difference between the third distribution variance and the second distribution variance of the final projected contour points is smaller than the second preset difference, that is, the overall distribution size of the final projected contour points is close to the size of the target detection frame.

[0122] In some embodiments, the second preset difference includes a third sub-preset difference and a fourth sub-preset difference.

[0123] In some embodiments, in response to the difference between the first sub-distribution variance and the third sub-distribution variance being greater than the third sub-preset difference, the point cloud object processing device increases or decreases the number of projection contour points to obtain the final projection contour points. It can be understood that the horizontal distribution size of the final projection contour points is similar to the width of the target detection box.

[0124] In some embodiments, in response to the difference between the second sub-distribution variance and the fourth sub-distribution variance being greater than the fourth sub-preset difference, the point cloud object processing device increases or decreases the number of projected contour points to obtain the final projected contour points. It can be understood that the vertical distribution size of the final projected contour points is close to the height of the target detection box.

[0125] S26, using the final projected contour point to determine the corresponding second target contour point from the point cloud frame.

[0126] In the above scheme, the first target contour point corresponding to the target in the point cloud frame is projected into the image frame at the same time to obtain the projection contour point, and the number of projection contour points is adjusted by comparing the positional relationship between the target detection frame corresponding to the target in the image frame and the projection contour point to obtain the final projection contour point, and then the second target contour point corresponding to the final projected contour point is determined from the point cloud frame. The point cloud target processing method provided in the present application optimizes the contour points of the target in the point cloud frame using the image detection results, so that the final second target contour point is more accurate, thereby improving the stability of subsequent operations of the intelligent driving car.

[0127] Furthermore, by adjusting the number of projected contour points, the distribution variance of the final projected contour points is made close to the variance of the target detection frame, thereby improving the accuracy of the overall distribution of the projected contour points.

[0128] Another embodiment of the method for processing a point cloud target provided by the present application may specifically include the following steps:

[0129] S31, determining the point cloud detection results and image detection results belonging to the same target from the point cloud frame and image frame at the same time.

[0130] S32, projecting the first target contour point into the image frame to obtain a projected contour point.

[0131] S33, obtaining a first distribution mean value and a first distribution variance of the projection contour points.

[0132] S34, obtaining a second distribution mean and a second distribution variance of the target detection frame.

[0133] S35 , in response to the difference between the first distribution mean and the second distribution mean being greater than or equal to the first preset difference, adjusting the number of projection contour points to obtain final projection contour points.

[0134] S36 , in response to the difference between the first distribution variance and the second distribution variance being greater than or equal to the second preset difference, adjusting the number of projection contour points to obtain final projection contour points.

[0135] Furthermore, the final projected contour points obtained in this embodiment are located at an overall position close to the overall position of the target detection frame, and the overall distribution size is close to the size of the target detection frame. That is, this embodiment adjusts the number of projected contour points so that the overall distribution of the final projected contour points is as similar as possible to the overall distribution of the target detection frame, thereby optimizing the overall position and overall size of the projected contour points.

[0136] It should be noted that S35 can be repeatedly executed, and S36 can also be repeatedly executed, so that the overall distribution of the final projection contour points is closer to the overall distribution of the target detection frame.

[0137] S37, using the final projected contour point to determine the corresponding second target contour point from the point cloud frame.

[0138] In the above scheme, the first target contour point corresponding to the target in the point cloud frame is projected into the image frame at the same time to obtain the projection contour point, and the number of projection contour points is adjusted by comparing the positional relationship between the target detection frame corresponding to the target in the image frame and the projection contour point to obtain the final projection contour point, and then the second target contour point corresponding to the final projected contour point is determined from the point cloud frame. The point cloud target processing method provided in the present application optimizes the contour points of the target in the point cloud frame using the image detection results, so that the final second target contour point is more accurate, thereby improving the stability of subsequent operations of the intelligent driving car.

[0139] Furthermore, by adjusting the number of projected contour points, the final distribution variance of the projected contour points is made close to the variance of the target detection frame, and the final distribution mean of the projected contour points is made close to the mean of the target detection frame, thereby improving the accuracy of the overall position and overall distribution of the projected contour points.

[0140] Another embodiment of the method for processing a point cloud target provided by the present application may specifically include the following steps:

[0141] S41, determining the point cloud detection result and the image detection result belonging to the same target from the point cloud frame and the image frame at the same time.

[0142] S42, projecting the first target contour point into the image frame to obtain a projected contour point.

[0143] S43, obtaining a first distribution mean of the projection contour points and / or a first distribution variance of the projection contour points.

[0144] S44, obtaining a second distribution mean and / or a second distribution variance of the target detection frame.

[0145] S45, determining a first missing direction and a first adjustment step length of the projected contour point relative to the target detection frame based on a difference between the first distribution mean and the second distribution mean.

[0146] In some embodiments, the point cloud target processing device calculates the absolute value of the difference between the first distribution mean and the second distribution mean to obtain a first difference value. The first adjustment step is positively correlated with the first difference value, that is, the larger the first difference value, the larger the first adjustment step; the smaller the first difference value, the smaller the first adjustment step. In the subsequent iterative adjustment process, the first distribution mean of the projected contour point is made closer to the second distribution mean of the target detection frame.

[0147] In some possible embodiments, the first distribution mean includes a first sub-distribution mean of the projected contour points in the horizontal direction, and / or a second sub-distribution mean of the projected contour points in the vertical direction.

[0148] In some possible embodiments, the second distribution mean includes a third sub-distribution mean of the target detection frame in the horizontal direction, and / or a fourth sub-distribution mean of the target detection frame in the vertical direction.

[0149] In some embodiments, if the first sub-distribution mean is greater than the third sub-distribution mean, the horizontal coordinate of the center of the projected contour point is greater than the horizontal coordinate of the center of the target detection frame, that is, the center of the projected contour point is further to the right relative to the center of the target detection frame. In this case, the first missing direction is horizontally to the left along the right plane of the target detection frame.

[0150] In some embodiments, if the first sub-distribution mean is smaller than the third sub-distribution mean, the horizontal coordinate of the center of the projected contour point is smaller than the horizontal coordinate of the center of the target detection frame, that is, the center of the projected contour point is further to the left relative to the center of the target detection frame. In this case, the first missing direction is horizontally to the right along the left plane of the target detection frame.

[0151] In some embodiments, if the second sub-distribution mean is greater than the fourth sub-distribution mean, the ordinate of the center of the projected contour point is greater than the ordinate of the center of the target detection frame, that is, the center of the projected contour point is higher than the center of the target detection frame. In this case, the first missing direction is vertically downward along the upper plane of the target detection frame.

[0152] In some embodiments, if the second sub-distribution mean is smaller than the fourth sub-distribution mean, the ordinate of the center of the projected contour point is smaller than the ordinate of the center of the target detection frame, that is, the center of the projected contour point is lower than the center of the target detection frame. In this case, the first missing direction is vertically upward along the lower plane of the target detection frame.

[0153] S46, in the point cloud frame, determine a first contour point sampling area along a first missing direction according to the first target contour point.

[0154] The size of the first contour point sampling area is determined by the first adjustment step.

[0155] In some embodiments, the point cloud detection result includes a point cloud detection box, wherein the point cloud detection box is within the point cloud frame.

[0156] In some embodiments, the first sub-distribution mean is greater than the third sub-distribution mean. Then the first missing direction is horizontally to the left along the right plane of the target detection frame, and the point cloud target processing device determines the first contour point sampling area on the left plane of the point cloud detection frame. Among them, the first contour point sampling area is a cuboid, the length of the first contour point sampling area is the first adjustment step, the width of the first contour point sampling area is the same as the width of the point cloud detection frame, the height of the first contour point sampling area is the same as the height of the point cloud detection frame, and in addition, the right plane of the first contour point sampling area is coplanar with the left plane of the point cloud detection frame.

[0157] In some embodiments, if the first sub-distribution mean is less than the third sub-distribution mean, the first missing direction is horizontally to the right along the left plane of the target detection frame, and the point cloud target processing device determines the first contour point sampling area on the right plane of the point cloud detection frame. The first contour point sampling area is a cuboid, the length of the first contour point sampling area is the first adjustment step, the width of the first contour point sampling area is the same as the width of the point cloud detection frame, the height of the first contour point sampling area is the same as the height of the point cloud detection frame, and the left plane of the first contour point sampling area is coplanar with the right plane of the point cloud detection frame.

[0158] In some embodiments, if the second sub-distribution mean is greater than the fourth sub-distribution mean, the first missing direction is vertically downward along the upper plane of the target detection frame, and the point cloud target processing device determines the first contour point sampling area on the lower plane of the point cloud detection frame. The first contour point sampling area is a cuboid, the height of the first contour point sampling area is the first adjustment step, the width of the first contour point sampling area is the same as the width of the point cloud detection frame, the length of the first contour point sampling area is the same as the length of the point cloud detection frame, and the upper plane of the first contour point sampling area is coplanar with the lower plane of the point cloud detection frame.

[0159] In some embodiments, if the second sub-distribution mean is less than the fourth sub-distribution mean, the first missing direction is vertically upward along the lower plane of the target detection frame, and the point cloud target processing device determines the first contour point sampling area on the upper plane of the point cloud detection frame. The first contour point sampling area is a cuboid, the height of the first contour point sampling area is the first adjustment step, the width of the first contour point sampling area is the same as the width of the point cloud detection frame, the length of the first contour point sampling area is the same as the length of the point cloud detection frame, and the lower plane of the first contour point sampling area is coplanar with the upper plane of the point cloud detection frame.

[0160] S47, sampling the data points in the first contour point sampling area and projecting them to the image frame to obtain first projected sampling points.

[0161] In some embodiments, the first contour point sampling area is located on the left or right side of the point cloud detection frame, and the point cloud target processing device collects a certain number of data points in the first contour point sampling area and projects these data points to the image frame to obtain first projected sampling points.

[0162] Among them, the number of collected data points satisfies the following relationship:

[0163] n1=(w / W)*N

[0164] Among them, n1 is the number of collected data points, w is the width of the first contour point sampling area, that is, the first adjustment step, W is the width of the point cloud detection box, and N is the number of projected contour points.

[0165] In some embodiments, the first contour point sampling area is located on the upper or lower side of the point cloud detection frame, and the point cloud target processing device collects a certain number of data points in the first contour point sampling area and projects these data points to the image frame to obtain first projected sampling points.

[0166] Among them, the number of collected data points satisfies the following relationship:

[0167] n2=(h / W)*N

[0168] Among them, n2 is the number of collected data points, h is the height of the first contour point sampling area, that is, the first adjustment step, H is the height of the point cloud detection box, and N is the number of projected contour points.

[0169] S48, using the first projection sampling point, adjusting the number of projection contour points to obtain final projection contour points.

[0170] In some embodiments, the point cloud object processing device adds the first projected sampling points to the projected contour points, and by increasing the number of projected contour points, the distribution mean of the adjusted projected contour points is further close to the second distribution mean of the target detection frame. Furthermore, after multiple iterations, the difference between the third distribution mean of the final projected contour points and the second distribution mean is less than the first preset difference.

[0171] S49, using the final projected contour point to determine the corresponding second target contour point from the point cloud frame.

[0172] In the above scheme, the first target contour point corresponding to the target in the point cloud frame is projected into the image frame at the same time to obtain the projection contour point, and the number of projection contour points is adjusted by comparing the positional relationship between the target detection frame corresponding to the target in the image frame and the projection contour point to obtain the final projection contour point, and then the second target contour point corresponding to the final projected contour point is determined from the point cloud frame. The point cloud target processing method provided in the present application optimizes the contour points of the target in the point cloud frame using the image detection results, so that the final second target contour point is more accurate, thereby improving the stability of subsequent operations of the intelligent driving car.

[0173] Furthermore, by adjusting the number of projection contour points, the distribution mean of the final projection contour points is made close to the mean of the target detection frame, thereby improving the accuracy of the overall position of the projection contour points.

[0174] Another embodiment of the method for processing a point cloud target provided by the present application may specifically include the following steps:

[0175] S51, determining the point cloud detection result and the image detection result belonging to the same target from the point cloud frame and the image frame at the same time.

[0176] S52, projecting the first target contour point into the image frame to obtain a projected contour point.

[0177] S53, obtaining a first distribution mean of the projection contour points and / or a first distribution variance of the projection contour points.

[0178] S54, obtaining a second distribution mean and / or a second distribution variance of the target detection frame.

[0179] S55, determining a second missing direction and a second adjustment step size based on the difference between the first distribution variance and the second distribution variance.

[0180] In some embodiments, the point cloud target processing device calculates the absolute value of the difference between the first distribution variance and the second distribution variance to obtain a second difference value. The second adjustment step is positively correlated with the second difference value, that is, the larger the second difference value, the larger the second adjustment step; the smaller the second difference value, the smaller the second adjustment step. In the subsequent iterative adjustment process, the first distribution variance of the projected contour point is made closer to the second distribution variance of the target detection frame.

[0181] In some possible embodiments, the first distribution variance includes a first sub-distribution variance of the projected contour points in a horizontal direction, and / or a second sub-distribution variance of the projected contour points in a vertical direction.

[0182] In some possible embodiments, the second distribution variance includes a third sub-distribution variance of the target detection frame in the horizontal direction, and / or a fourth sub-distribution variance of the target detection frame in the vertical direction.

[0183] In some embodiments, the variance of the first sub-distribution is greater than the variance of the third sub-distribution, that is, the horizontal distribution of the projected contour points is greater than the distribution of the target detection frame, that is, the distribution width of the projected contour points is greater than the width of the target detection frame. In this case, the second missing direction is from the horizontal outer side of the projected contour points to the center of the projected contour points.

[0184] In some embodiments, the variance of the first sub-distribution is smaller than the variance of the third sub-distribution, that is, the horizontal distribution of the projected contour points is smaller than the distribution of the target detection frame, that is, the distribution width of the projected contour points is smaller than the width of the target detection frame. In this case, the second missing direction is from the center of the projected contour points horizontally to the horizontal outside of the projected contour points.

[0185] In some embodiments, the variance of the second sub-distribution is greater than the variance of the fourth sub-distribution, that is, the distribution of the projected contour points in the vertical direction is greater than the distribution of the target detection frame, that is, the height of the entire distribution of the projected contour points is greater than the height of the target detection frame. In this case, the second missing direction is the vertical outer side of the projected contour points pointing vertically to the center of the projected contour points.

[0186] In some embodiments, the variance of the second sub-distribution is smaller than the variance of the fourth sub-distribution, that is, the distribution of the projected contour points in the vertical direction is smaller than the distribution of the target detection frame, that is, the distribution height of the projected contour points is smaller than the height of the target detection frame. In this case, the second missing direction is from the center of the projected contour point vertically to the vertical outside of the projected contour point.

[0187] There is at least one second missing direction.

[0188] S56, adjusting the number of projection contour points by using the second missing direction and the second adjustment step length to obtain final projection contour points.

[0189] In some embodiments, the distribution height of the projected contour points is greater than the height of the target detection frame, and / or the distribution width of the projected contour points is greater than the width of the target detection frame. The point cloud target processing device uses the second missing direction and the second adjustment step size to determine the target area within the projected contour points, and deletes the projected contour points in the target area, that is, reduces the number of projected contour points on both sides to reduce the distribution height and / or distribution width of the projected contour points.

[0190] In some embodiments, the distribution height of the projected contour points is less than the height of the target detection frame, and / or the distribution width of the projected contour points is less than the width of the target detection frame. The point cloud target processing device uses the second missing direction and the second adjustment step size to determine the target area outside the projected contour points, and samples and projects data points in the point cloud area corresponding to the target area to obtain newly added projected contour points, and adds the newly added projected contour points to the projected contour points, that is, increases the number of projected contour points on both sides to increase the distribution height and / or distribution width of the projected contour points.

[0191] S57, using the final projected contour point to determine the corresponding second target contour point from the point cloud frame.

[0192] In the above scheme, the first target contour point corresponding to the target in the point cloud frame is projected into the image frame at the same time to obtain the projection contour point, and the number of projection contour points is adjusted by comparing the positional relationship between the target detection frame corresponding to the target in the image frame and the projection contour point to obtain the final projection contour point, and then the second target contour point corresponding to the final projected contour point is determined from the point cloud frame. The point cloud target processing method provided in the present application optimizes the contour points of the target in the point cloud frame using the image detection results, so that the final second target contour point is more accurate, thereby improving the stability of subsequent operations of the intelligent driving car.

[0193] Furthermore, by adjusting the number of projected contour points, the distribution variance of the final projected contour points is made close to the variance of the target detection frame, thereby improving the accuracy of the overall distribution of the projected contour points.

[0194] Another embodiment of the method for processing a point cloud target provided by the present application may specifically include the following steps:

[0195] S61, determining the point cloud detection results and image detection results belonging to the same target from the point cloud frame and the image frame at the same time.

[0196] S62, projecting the first target contour point into the image frame to obtain a projected contour point.

[0197] S63, obtaining a first distribution mean of the projection contour points and / or a first distribution variance of the projection contour points.

[0198] S64, obtaining a second distribution mean and / or a second distribution variance of the target detection frame.

[0199] S65, determining a second missing direction and a second adjustment step size based on a difference between the first distribution variance and the second distribution variance.

[0200] The second missing direction is the missing direction of the projected contour point relative to the target detection frame.

[0201] S66, in the point cloud frame, determine a second contour point sampling area along a second missing direction according to the first target contour point.

[0202] The size of the second contour point sampling area is determined by the second adjustment step.

[0203] In some embodiments, the point cloud detection result includes a point cloud detection box.

[0204] In some embodiments, the point cloud detection result includes a point cloud detection box, wherein the point cloud detection box is within the point cloud frame.

[0205] In some embodiments, the variance of the first sub-distribution is greater than the variance of the third sub-distribution, that is, the second missing direction is horizontally pointing from the horizontal outer side of the projected contour point to the center of the projected contour point, and the point cloud detection device determines at least one second contour point sampling area in the point cloud detection frame. The second contour point sampling area is a cuboid, the width of the second contour point sampling area is the same as the width of the point cloud detection frame, and the height of the second contour point sampling area is the same as the height of the point cloud detection frame. At this time, the second missing direction is horizontally to the right and / or horizontally to the left.

[0206] In some possible embodiments, the number of the second contour point sampling areas is 1, and the length of the second contour point sampling areas is the second adjustment step. In some possible embodiments, the number of the second contour point sampling areas is 2, and the length of the second contour point sampling areas is 0.5 times the second adjustment step, and the two second contour point sampling areas are symmetrical when viewed from a direction perpendicular to a plane formed by the length and width of the point cloud detection frame.

[0207] In some embodiments, the variance of the first sub-distribution is smaller than the variance of the third sub-distribution, that is, the second missing direction is the horizontal direction from the center of the projected contour point to the horizontal outside of the projected contour point, and the point cloud detection device determines at least one second contour point sampling area outside the point cloud detection frame. Among them, the second contour point sampling area is a cuboid, the width of the second contour point sampling area is the same as the width of the point cloud detection frame, and the height of the second contour point sampling area is the same as the height of the point cloud detection frame. At this time, the second missing direction is horizontal to the right and / or horizontal to the left. In addition, the left plane of the second contour point is coplanar with the right plane of the point cloud detection frame, and / or, the right plane of the second contour point is coplanar with the left plane of the point cloud detection frame.

[0208] In some possible embodiments, the number of the second contour point sampling areas is 1, and the length of the second contour point sampling areas is the second adjustment step. In some possible embodiments, the number of the second contour point sampling areas is 2, and the length of the second contour point sampling areas is 0.5 times the second adjustment step, and the two second contour point sampling areas are symmetrical when viewed from a direction perpendicular to a plane formed by the length and width of the point cloud detection frame.

[0209] In some embodiments, the variance of the second sub-distribution is greater than the variance of the fourth sub-distribution, that is, the second missing direction is vertically pointing from the vertical outside of the projected contour point to the center of the projected contour point, and the point cloud detection device determines at least one second contour point sampling area in the point cloud detection frame. The second contour point sampling area is a cuboid, the width of the second contour point sampling area is the same as the width of the point cloud detection frame, and the length of the second contour point sampling area is the same as the length of the point cloud detection frame. At this time, the second missing direction is vertically upward and / or vertically downward.

[0210] In some possible embodiments, the number of the second contour point sampling areas is 1, and the height of the second contour point sampling areas is the second adjustment step. In some possible embodiments, the number of the second contour point sampling areas is 2, and the height of the second contour point sampling areas is 0.5 times the second adjustment step, and the two second contour point sampling areas are symmetrical when viewed from a direction perpendicular to a plane formed by the length and width of the point cloud detection frame.

[0211] In some embodiments, the variance of the second sub-distribution is smaller than the variance of the fourth sub-distribution, that is, the second missing direction is from the center of the projected contour point vertically to the vertical outside of the projected contour point, and the point cloud detection device determines at least one second contour point sampling area outside the point cloud detection frame. The second contour point sampling area is a cuboid, the length of the second contour point sampling area is the same as the length of the point cloud detection frame, and the width of the second contour point sampling area is the same as the width of the point cloud detection frame. At this time, the second missing direction is vertically upward and / or vertically downward.

[0212] In some possible embodiments, the number of the second contour point sampling areas is 1, and the height of the second contour point sampling areas is the second adjustment step. In some possible embodiments, the number of the second contour point sampling areas is 2, and the height of the second contour point sampling areas is 0.5 times the second adjustment step, and the two second contour point sampling areas are symmetrical when viewed from a direction perpendicular to a plane formed by the length and width of the point cloud detection frame.

[0213] S67, sampling the data points in the second contour point sampling area and projecting them into the image frame to obtain second projected sampling points.

[0214] In some possible embodiments, the second contour point sampling area is outside the point cloud detection frame, and the point cloud target processing device collects a certain number of data points in the second contour point sampling area and projects these data points to the image frame to obtain second projected sampling points.

[0215] In some embodiments, the number of collected data points satisfies the following relationship:

[0216] n3=(w' / W)*N

[0217] Wherein, n1 is the number of collected data points, w' is the width of the second contour point sampling area, that is, the second adjustment step, W is the width of the point cloud detection box, and N is the number of projected contour points.

[0218] In some embodiments, the number of collected data points satisfies the following relationship:

[0219] n4=(h' / H)*N

[0220] Wherein, n1 is the number of collected data points, h' is the height of the second contour point sampling area, that is, the second adjustment step, H is the height of the point cloud detection box, and N is the number of projected contour points.

[0221] In some possible embodiments, the second contour point sampling area is within the point cloud detection frame, and the point cloud object processing device collects part of the first projection contour points in the second contour point sampling area as second projection sampling points.

[0222] S68, using the second projection sampling points, adjusting the number of projection contour points to obtain final projection contour points.

[0223] In some embodiments, the second contour point sampling area is outside the point cloud detection frame, and the point cloud target processing device adds the second projection sampling point to the projection contour point, and increases the distribution variance of the adjusted projection contour point by adding the number of projection contour points, so that it is further close to the second variance of the target detection frame. Further, after multiple iterations, the difference between the third distribution variance of the final projection contour point and the second distribution variance is less than the second preset difference.

[0224] In some embodiments, the second contour point sampling area is within the point cloud detection frame, and the point cloud detection device deletes the second projection sampling point from the projection contour point, and reduces the distribution variance of the adjusted projection contour point by reducing the number of projection contour points, so that it is further close to the second variance of the target detection frame. Further, after multiple iterations, the difference between the third distribution variance of the final projection contour point and the second distribution variance is less than the second preset difference.

[0225] S69, using the final projected contour point to determine the corresponding second target contour point from the point cloud frame.

[0226] In the above scheme, the first target contour point corresponding to the target in the point cloud frame is projected into the image frame at the same time to obtain the projection contour point, and the number of projection contour points is adjusted by comparing the positional relationship between the target detection frame corresponding to the target in the image frame and the projection contour point to obtain the final projection contour point, and then the second target contour point corresponding to the final projected contour point is determined from the point cloud frame. The point cloud target processing method provided in the present application optimizes the contour points of the target in the point cloud frame using the image detection results, so that the final second target contour point is more accurate, thereby improving the stability of subsequent operations of the intelligent driving car.

[0227] Furthermore, by adjusting the number of projected contour points, the distribution variance of the final projected contour points is made close to the variance of the target detection frame, thereby improving the accuracy of the overall distribution of the projected contour points.

[0228] Another embodiment of the method for processing a point cloud target provided by the present application may specifically include the following steps:

[0229] S71, determining the point cloud detection results and image detection results belonging to the same target from the point cloud frame and the image frame at the same time.

[0230] S72, projecting the first target contour point into the image frame to obtain a projected contour point.

[0231] S73, obtaining a first distribution mean of the projection contour points and / or a first distribution variance of the projection contour points.

[0232] S74, obtaining a second distribution mean and / or a second distribution variance of the target detection frame.

[0233] S75, in response to the first distribution variance being greater than the second distribution variance, and the difference between the first distribution mean and the second distribution mean being less than a first preset difference, determining a second missing direction and a second adjustment step size of the target detection frame relative to the projection contour point.

[0234] In this embodiment, the difference between the first distribution variance and the second distribution variance is greater than or equal to the second preset difference.

[0235] It can be understood that in this embodiment, the first distribution variance is greater than the second distribution variance, and the difference between the first distribution mean and the second distribution mean is less than the first preset difference, that is, the center point of the projected contour point is close to the center of the target detection frame, and the distribution size of the projected contour point is greater than the size of the target detection frame. In other words, this embodiment needs to delete the projection points outside the projection contour point.

[0236] Specifically, the first distribution variance includes a first sub-distribution variance of the projected contour point in the horizontal direction, and / or a second sub-distribution variance of the projected contour point in the vertical direction. The second distribution variance includes a third sub-distribution variance of the target detection frame in the horizontal direction, and / or a fourth sub-distribution variance of the target detection frame in the vertical direction.

[0237] In some embodiments, the variance of the first sub-distribution is greater than the variance of the third sub-distribution. In this case, the second missing direction is the horizontal outer side of the projected contour point pointing horizontally to the center of the projected contour point, that is, the second missing direction is horizontal to the left and / or horizontal to the right. Further, the point cloud target processing device takes the left plane of the point cloud detection frame as the starting point, translates the second translation step to the right, and determines the first reference plane, and / or takes the right plane of the point cloud detection frame as the starting point, translates the second translation step to the left, and determines the second reference plane. It can be understood that the point cloud target processing device projects the first reference plane into the image frame to obtain the first reference line, and similarly, obtains the second reference line in the image frame.

[0238] In some embodiments, the variance of the second sub-distribution is greater than the variance of the fourth sub-distribution. In this case, the second missing direction is the vertical outer side of the projected contour point pointing vertically to the center of the projected contour point, that is, the second missing direction is vertically upward and / or vertically downward. Further, the point cloud target processing device takes the upper plane of the point cloud detection frame as the starting point, translates downward by the second translation step, and determines the third reference plane, and / or, takes the lower plane of the point cloud detection frame as the starting point, translates upward by the second translation step, and determines the fourth reference plane. It can be understood that the point cloud target processing device projects the third reference plane into the image frame to obtain the third reference line, and similarly, obtains the fourth reference line in the image frame.

[0239] S76, along the opposite direction of the second missing direction, using the second adjustment step length, delete the projection contour points located outside the preset distance from the center point of the target detection frame to obtain the remaining projection contour points.

[0240] In some embodiments, the second missing direction is horizontally to the left and / or horizontally to the right. Further, the point cloud object processing device deletes the projection contour points located to the left of the first reference line and / or deletes the projection contour points located to the right of the second reference line to obtain the remaining projection contour points.

[0241] Wherein, the distance between the first reference line and the center point of the target detection frame is a preset distance. And / or, the distance between the second reference line and the center point of the target detection frame is a preset distance.

[0242] In some embodiments, the second missing direction is vertically upward and / or vertically downward. Further, the point cloud object processing device deletes the projection contour points located above the third reference line and / or deletes the projection contour points located below the fourth reference line to obtain the remaining projection contour points.

[0243] The distance between the third reference line and the center point of the target detection frame is a preset distance. And / or the distance between the fourth reference line and the center point of the target detection frame is a preset distance.

[0244] S77, in the point cloud frame, along the second missing direction, with the distance between the boundary of the target detection frame and the preset distance as a reference, determine a second contour point sampling area.

[0245] In some embodiments, the second missing direction is horizontally to the left and / or horizontally to the right, and the point cloud object processing device uses the first reference plane as the reference plane, and determines a second contour point sampling area along the horizontal right direction with a length of the second missing direction, wherein the left plane of the second contour point sampling area is coplanar with the first reference plane, and / or the point cloud object processing device uses the second reference plane as the reference plane, and determines a second contour point sampling area along the horizontal left direction with a length of the second missing direction, wherein the right plane of the second contour point sampling area is coplanar with the second reference plane.

[0246] In some embodiments, the second missing direction is vertically upward and / or vertically downward, and the point cloud target processing device uses the third reference plane as the reference plane, and determines the second contour point sampling area along the vertical downward direction with a height of the second missing direction, wherein the upper plane of the second contour point sampling area is coplanar with the third reference plane, and / or the point cloud target processing device uses the fourth reference plane as the reference plane, and determines the second contour point sampling area along the vertical upward direction with a height of the second missing direction, wherein the lower plane of the second contour point sampling area is coplanar with the fourth reference plane.

[0247] S78, sampling the data points in the second contour point sampling area, and projecting them into the image frame to obtain second projected sampling points.

[0248] S79, adding the second projection sampling point to the remaining projection contour points to obtain the final projection contour points.

[0249] It should be noted that S76 deletes some projection contour points, and the remaining projection contour points may have the problem of insufficient number of projection points, which affects the overall distribution of the remaining projection contour points. Based on this, by sampling and projecting data points in the second contour point sampling area, the remaining projection contour points are supplemented with projection points to avoid the problem of insufficient number of remaining projection contour points.

[0250] S80, using the final projected contour point to determine the corresponding second target contour point from the point cloud frame.

[0251] In the above scheme, the first target contour point corresponding to the target in the point cloud frame is projected into the image frame at the same time to obtain the projection contour point, and the number of projection contour points is adjusted by comparing the positional relationship between the target detection frame corresponding to the target in the image frame and the projection contour point to obtain the final projection contour point, and then the second target contour point corresponding to the final projected contour point is determined from the point cloud frame. The point cloud target processing method provided in the present application optimizes the contour points of the target in the point cloud frame using the image detection results, so that the final second target contour point is more accurate, thereby improving the stability of subsequent operations of the intelligent driving car.

[0252] Furthermore, by adjusting the number of projected contour points, the distribution variance of the final projected contour points is made close to the variance of the target detection frame, thereby improving the accuracy of the overall distribution of the projected contour points.

[0253] Another embodiment of the method for processing a point cloud target provided by the present application may specifically include the following steps:

[0254] S81, determining the point cloud detection results and image detection results belonging to the same target from the point cloud frame and image frame at the same time.

[0255] The point cloud detection result includes a first target contour point corresponding to the target, and the image detection result includes a target detection box corresponding to the target.

[0256] S82, obtaining a projection matrix between the point cloud frame and the image frame.

[0257] In some embodiments, the position of the data point in the point cloud frame is the position in the world coordinate system, and the point cloud object processing device obtains the pose transformation matrix between the radar sensor and the camera, and obtains the camera intrinsic parameters. Further, the point cloud object processing device determines the projection matrix based on the pose transformation matrix and the camera intrinsic parameters.

[0258] S83, based on the projection matrix, project the first target contour point from the point cloud frame to the image frame to obtain a projected contour point.

[0259] Specifically, the point cloud target processing device determines the position of each data point in the image frame according to the projection matrix, and based on this, projects the first target contour point from the point cloud frame to the image frame to obtain the projected contour point.

[0260] S84, adjusting the number of projection contour points according to the positional relationship between the projection contour points and the target detection frame to obtain final projection contour points.

[0261] S85, using the final projected contour point to determine the corresponding second target contour point from the point cloud frame.

[0262] In the above scheme, the first target contour point corresponding to the target in the point cloud frame is projected into the image frame at the same time to obtain the projection contour point, and the number of projection contour points is adjusted by comparing the positional relationship between the target detection frame corresponding to the target in the image frame and the projection contour point to obtain the final projection contour point, and then the second target contour point corresponding to the final projected contour point is determined from the point cloud frame. The point cloud target processing method provided in the present application optimizes the contour points of the target in the point cloud frame using the image detection results, so that the final second target contour point is more accurate, thereby improving the stability of subsequent operations of the intelligent driving car.

[0263] See also Figure 2 , Figure 2 It is a flowchart of another embodiment of the method for processing point cloud targets provided in the present application.

[0264] like Figure 2 As shown, another embodiment of the method for processing a point cloud target provided by the present application may specifically include the following steps:

[0265] S101, data acquisition.

[0266] This step requires obtaining the target detection results from the original laser radar point cloud data and image data. The laser point cloud passes through a point cloud detection network or a point cloud segmentation network, such as pointpillar, pointnet, etc., to obtain a 3D target detection frame and the contour point data corresponding to the target. The image data passes through a visual detection network, such as a yolo network, to obtain a 2D detection frame on the image.

[0267] S102, contour point projection.

[0268] There are usually two ways to fuse laser data and image data. One is to obtain 3D information from a 2D detection frame, such as visual depth estimation. However, the current visual depth estimation effect is still far behind the detection results of lidar and millimeter waves. The other is to project the 3D information onto a 2D image. This method relies on the establishment of a projection matrix. Through a mature calibration method, an accurate projection matrix can be obtained. This step utilizes the projection from 3D to 2D, and the depth information of the projection point will still be retained after projection to facilitate subsequent resampling of the 3D layer. Furthermore, the contour points after projection are the projection contour points mentioned above.

[0269] S103, calculating data distribution.

[0270] After completing the projection of the 3D contour points onto the 2D image, the point cloud object processing device obtains the projected contour points corresponding to the contour points and calculates the distribution of the projected contour points in the image frame.

[0271] In this embodiment, it is assumed that the projection contour points obey Gaussian distribution. Based on this, the point cloud object processing device can obtain the distribution N of the projection points.

[0272] Similarly, for the 2D object detection box, it is also considered to obey N(μ,σ 2 ), where is a two-dimensional vector representing the center point of the target detection box. It is a diagonal function, and the diagonal elements represent the width and height information of the target detection frame. Thus, the point cloud target processing device can obtain the distribution of the projection points and the target detection frame.

[0273] like Figure 3 As shown in the figure, the left side is the 3D detection frame of the laser, where the dots represent the contour points of the laser in the current frame. The right side is the image data, and the box in the image data is the image detection frame. The black dots in the box represent the projection points of the contour points. Since it has been assumed to be a Gaussian distribution, it is necessary to obtain the mean and variance of the Gaussian distribution of the two. Finally, the Gaussian distribution of the final projected contour points and the target detection frame can be obtained.

[0274] S104, contour point resampling.

[0275] In this embodiment, the point cloud target processing device calculates the deviation between the mean of the projected contour points and the mean of the target detection frame, and calculates the deviation between the variance of the projected contour points and the variance of the target detection frame. Figure 3As shown in the figure, for the x direction (i.e. the horizontal coordinate in the image), the distribution mean μ′ of the projection points is biased to the right compared to the mean μ of the detection box, and the distribution variance σ′ of the projection points is smaller than the variance σ of the detection box, which indicates that there are missing contour points on the left side of the 3D contour points (in the opposite direction of the offset). Then the optimization direction is to add more sampling points to the left side of the target.

[0276] The purpose of resampling is to compare these two distributions and determine how to sample the missing contour points in the 3D layer in the next step.

[0277] In some embodiments, the point cloud object processing device first calculates the mean and variance deviation of the two distributions to obtain the direction of the next sampling. This direction is the opposite direction of the deviation between the contour point distribution and the detection box distribution. Figure 3 , the next sampling direction is to the left. After the direction is determined, the width is expanded along this direction with a variable step length to obtain a 3D expansion frame. Then sampling is performed in the expansion frame in proportion, and the sampling ratio needs to take into account the ratio of the expansion width to the total width. Finally, reprojection is performed based on the new contour points, and S102-S104 are repeated. Until the condition that the distribution of the two is roughly the same is met, the obstacle size value at this time is output.

[0278] In some embodiments, the point cloud object processing device performs resampling in the expansion frame, and the number of sampling points satisfies the following relationship:

[0279] Number of sampling points = (expanded width / total width)*total number of contour points.

[0280] In some possible embodiments, the distribution variance of the projected contour points is similar to the distribution variance of the target detection frame, and the lateral distribution mean of the projected contour points is greater than the lateral distribution mean of the target detection frame, that is, the size of the projected contour points is similar to the target detection frame, and the center of the projected contour points is located to the right of the center of the target detection frame, and the point cloud target processing device determines that the direction of the next sampling is horizontally to the left. Repeating the sampling and reprojection steps can be understood as the projected contour points gradually approaching the target detection frame.

[0281] The above scheme projects the first target contour point corresponding to the target in the point cloud frame to the image frame at the same time to obtain the projection contour point, and adjusts the number of projection contour points by comparing the positional relationship between the target detection frame corresponding to the target in the image frame and the projection contour point to obtain the final projection contour point, and then determines the second target contour point corresponding to the final projection contour point from the point cloud frame. The point cloud target processing method provided in the present application optimizes the contour points of the target in the point cloud frame using the image detection results, so that the final second target contour point is more accurate, thereby improving the stability of subsequent operations of the intelligent driving car.

[0282] In addition, although this scheme requires resampling of contour points, compared with using multi-frame point cloud data to redetermine the target contour points corresponding to the target, the amount of data processed is smaller, and the method based on a single frame (image frame, point cloud frame) has better applicability.

[0283] Please continue to see Figure 4 , Figure 4 The terminal device 500 of the embodiment of the present application includes a processor 51 and a memory 52 .

[0284] The processor 51 and the memory 52 are connected to the bus. The memory 52 stores program data. The processor 51 is used to execute the program data to implement the point cloud target processing method described in the above embodiment.

[0285] In the embodiment of the present application, the processor 51 may also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip having the ability to process signals. The processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 51 may also be any conventional processor, etc.

[0286] This application also provides a computer storage medium, please continue to refer to Figure 5 , Figure 5 It is a structural diagram of an embodiment of a computer storage medium provided in the present application. The computer storage medium 600 stores program data 61. When the program data 61 is executed by the processor, it is used to implement the point cloud target processing method of the above embodiment.

[0287] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.

[0288] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Equivalent structures or equivalent process changes made by utilizing the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for processing a point cloud target, characterized in that: The method comprises: Determine a point cloud detection result and an image detection result belonging to the same target from the point cloud frame and the image frame at the same time, wherein the point cloud detection result includes a first target contour point corresponding to the target, and the image detection result includes a target detection frame corresponding to the target; Projecting the first target contour point into the image frame to obtain a projected contour point; According to the positional relationship between the projection contour points and the target detection frame, the number of the projection contour points is adjusted to obtain final projection contour points; The final projected contour points are used to determine corresponding second target contour points from the point cloud frame.

2. The method for processing a point cloud target according to claim 1, characterized in that: The step of adjusting the number of the projection contour points according to the positional relationship between the projection contour points and the target detection frame to obtain the final projection contour points includes: Acquire a first distribution mean of the projection contour points and / or a first distribution variance of the projection contour points; and a second distribution mean and / or a second distribution variance of the target detection frame; In response to a difference between the first distribution mean and the second distribution mean being greater than or equal to a first preset difference, adjusting the number of the projection contour points to obtain the final projection contour points; and / or, In response to the difference between the first distribution variance and the second distribution variance being greater than or equal to a second preset difference, the number of the projection contour points is adjusted to obtain the final projection contour points.

3. The method for processing a point cloud target according to claim 2, characterized in that: In response to the absolute value of the difference between the first distribution mean and the second distribution mean being greater than or equal to a first preset difference, adjusting the number of the projection contour points to obtain the final projection contour points comprises: Determining a first missing direction and a first adjustment step size of the projected contour point relative to the target detection frame based on a difference between the first distribution mean and the second distribution mean; In the point cloud frame, according to the first target contour point, a first contour point sampling area is determined along the first missing direction, wherein a size of the first contour point sampling area is determined by the first adjustment step size; Sampling data points in the first contour point sampling area and projecting the data points to the image frame to obtain first projected sampling points; The number of the projection contour points is adjusted using the first projection sampling points to obtain the final projection contour points.

4. The method for processing a point cloud target according to claim 2, characterized in that: In response to the absolute value of the difference between the first distribution variance and the second distribution variance being greater than or equal to a second preset difference, adjusting the number of the projection contour points to obtain the final projection contour points includes: Determining a second missing direction and a second adjustment step size based on a difference between the first distribution variance and the second distribution variance; The second missing direction and the second adjustment step length are used to adjust the number of the projection contour points to obtain the final projection contour points.

5. The method for processing a point cloud target according to claim 4, characterized in that: The second missing direction is the missing direction of the projected contour point relative to the target detection frame; The adjusting the number of the projection contour points by using the second missing direction and the second adjustment step to obtain the final projection contour points includes: In the point cloud frame, according to the first target contour point, a second contour point sampling area is determined along the second missing direction, wherein the size of the second contour point sampling area is determined by the second adjustment step size; Sampling data points in the second contour point sampling area and projecting them into the image frame to obtain second projected sampling points; The second projection sampling points are used to adjust the number of the projection contour points to obtain the final projection contour points.

6. The method for processing a point cloud target according to claim 4, characterized in that: The determining a second missing direction and a second adjustment step size based on a difference between the first distribution variance and the second distribution variance comprises: In response to the first distribution variance being greater than the second distribution variance, and the difference between the first distribution mean and the second distribution mean being less than the first preset difference, determining a second missing direction and a second adjustment step length of the target detection frame relative to the projected contour point; The adjusting the number of the projection contour points by using the second missing direction and the second adjustment step to obtain the final projection contour points includes: Along the opposite direction of the second missing direction, using the second adjustment step size, the projected contour points located outside the preset distance from the center point of the target detection frame are deleted to obtain remaining projected contour points; In the point cloud frame, along the second missing direction, taking the distance between the boundary of the target detection frame and the preset distance as a reference, determining a second contour point sampling area; Sampling data points in the second contour point sampling area and projecting them into the image frame to obtain second projected sampling points; The second projection sampling points are added to the remaining projection contour points to obtain the final projection contour points.

7. The method for processing a point cloud target according to any one of claims 2 to 6, characterized in that: The first distribution mean includes a first sub-distribution mean of the projected contour points in the horizontal direction, and / or a second sub-distribution mean of the projected contour points in the vertical direction; The second distribution mean includes a third sub-distribution mean of the target detection frame in the horizontal direction, and / or a fourth sub-distribution mean of the target detection frame in the vertical direction; The first distribution variance includes a first sub-distribution variance of the projected contour points in a horizontal direction, and / or a second sub-distribution variance of the projected contour points in a vertical direction; The second distribution variance includes a third sub-distribution variance of the target detection frame in a horizontal direction, and / or a fourth sub-distribution variance of the target detection frame in a vertical direction.

8. The method for processing a point cloud target according to claim 1, characterized in that: The step of projecting the first target contour point into the image frame to obtain a projected contour point includes: Acquire a projection matrix between the point cloud frame and the image frame; Based on the projection matrix, the first target contour points are projected from the point cloud frame to the image frame to obtain projected contour points.

9. A terminal device, characterized in that: The terminal device includes a processor and a memory connected to the processor, wherein: The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the point cloud target processing method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The storage medium stores program instructions, and when the program instructions are executed, the method for processing a point cloud target according to any one of claims 1 to 8 is implemented.