3D target detection methods, systems and vehicles

Abstract

This invention provides a 3D target detection method, system, and vehicle, belonging to the field of target detection. The method includes: acquiring the 2D bounding box of the target in a first 2D image; establishing a projection relationship between the first 2D image and a point cloud; determining the 2D pixels corresponding to each point in the 3D point cloud and the 3D information corresponding to the 2D pixels based on the projection relationship, thereby obtaining a second 2D image containing 3D information, where the 3D information includes the depth value of the pixels; determining an extended bounding box in the second 2D image based on the 2D bounding box; acquiring the 3D information of all pixels within the extended bounding box; and fitting the 3D information of all pixels to obtain the 3D information of the target. This invention enables real-time computational detection of 3D targets.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a 3D target detection method, system, and vehicle. Background Technology

[0002] Currently, the main 3D (three-dimensional) object detection algorithms used in autonomous driving include: 1. Point cloud-based 3D object detection algorithms; 2. Binocular camera and RGB-D camera-based 3D object detection algorithms; and 3. Image and point cloud fusion-based 3D object detection algorithms. Point cloud-based 3D object detection algorithms lack color and texture information of the target, resulting in lower detection accuracy compared to image-based algorithms. Image-based algorithms, lacking distance information from the camera, often produce large errors in obtaining distance information. Therefore, image and point cloud fusion-based 3D object detection algorithms can obtain both color and texture information of the target, as well as distance information, leading to more accurate 3D object detection results.

[0003] 3D target detection algorithms based on point cloud and image fusion work by establishing a projection relationship between the point cloud and the image based on the calibrated intrinsic and extrinsic parameters of the two. However, projecting 3D point cloud data into an image can lead to point cloud information loss and target occlusion. Projecting image information into a 3D coordinate system also presents a problem: distant, large targets and near, small targets appear to have the same image size in the image. Therefore, 3D target detection algorithms based on the joint use of image and point cloud data warrant significant research.

[0004] Current point cloud and image feature matching algorithms primarily focus on using deep learning models to achieve feature matching between point clouds and images. This approach requires extensive data annotation, and model training is time-consuming. Furthermore, the model training results have weak generalization ability across different environments, leading to significant differences in application performance. Additionally, point cloud-based deep learning algorithms consume large amounts of resources in their computational units, are computationally complex, and cannot achieve real-time performance. Summary of the Invention

[0005] This invention provides a 3D target detection method, system, and vehicle. By designing a matching and fusion approach for image and point cloud data, 3D target detection is achieved. The algorithm is simple, has low computational complexity, and can realize real-time computational detection of 3D targets.

[0006] This invention provides a 3D target detection method, comprising:

[0007] Obtain the 2D bounding box of the target in the first 2D image;

[0008] Establish the projection relationship between the first 2D image and the point cloud, determine the 2D pixel corresponding to each point cloud in the 3D point cloud and the 3D information corresponding to the 2D pixel according to the projection relationship, and obtain a second 2D image containing 3D information, wherein the 3D information includes the depth value of the pixel.

[0009] Determine the expanded bounding box in the second 2D image based on the 2D bounding box;

[0010] Obtain 3D information of all pixels within the extended bounding box;

[0011] The 3D information of all the pixels is fitted to obtain the 3D information of the target under test.

[0012] Optionally, fitting the 3D information of all the pixels to obtain the 3D information of the target to be tested includes:

[0013] A histogram is determined based on the 3D information of all the pixels, wherein the horizontal axis of the histogram is the depth value of the pixel, and the vertical axis of the histogram is the number of pixels.

[0014] Determine the x-axis value corresponding to the peak value of the histogram;

[0015] Obtain 3D information of pixels within a preset range of the horizontal coordinate value; the preset range of the horizontal coordinate value is a preset distance range extended to both sides based on the horizontal coordinate value;

[0016] The 3D information of the target to be tested is determined based on the 3D information of the pixels within the preset range of the horizontal coordinate value.

[0017] Optionally, establishing the projection relationship between the first 2D image and the point cloud, determining the 2D pixels corresponding to each point cloud in the 3D point cloud based on the projection relationship, and the 3D information corresponding to the 2D pixels, to obtain a second 2D image containing 3D information, includes:

[0018] The intrinsic and extrinsic parameters of the 3D point cloud projected onto the first 2D image are determined by calibration;

[0019] A storage container for 3D information is constructed. The storage container is a set of storage spaces for each pixel, indexed by the pixels of the first 2D image, for storing the 3D information of the pixels.

[0020] The 2D pixel points corresponding to each point cloud in the 3D point cloud are determined based on the intrinsic and extrinsic parameters.

[0021] The 3D information corresponding to the 2D pixel is determined based on the storage container to obtain a second 2D image containing the 3D information.

[0022] Optionally, determining the expanded bounding box in the second 2D image based on the 2D bounding box includes:

[0023] Obtain the coordinates of the center point of the 2D bounding box;

[0024] In the second 2D image, the expanded bounding box is obtained by expanding the 2D bounding box by a preset ratio in both the width and height directions according to the coordinates of the center point.

[0025] Optionally, determining the x-coordinate value corresponding to the peak value of the histogram includes:

[0026] Determine the number of peaks in the histogram; when the number of peaks in the histogram is equal to 1, determine the x-coordinate value corresponding to the peak value of the histogram; when the number of peaks in the histogram is greater than 1, determine the x-coordinate value corresponding to each peak value of the histogram.

[0027] Optionally,

[0028] When the number of peaks in the histogram is greater than 1, the 3D information of the target to be measured is determined based on the 3D information of pixels within a preset range of the horizontal coordinate values, including:

[0029] Determine the x-axis value corresponding to each peak of the histogram;

[0030] Define the point cloud corresponding to all pixels within a preset range of each horizontal coordinate value as a point cloud cluster, and obtain the 3D information of each point cloud cluster;

[0031] The target point cloud cluster is determined based on the 3D information of each point cloud cluster;

[0032] The target point cloud cluster is fitted to obtain the 3D information of the target under test.

[0033] Optionally, determining the target point cloud cluster based on the 3D information of each point cloud cluster includes:

[0034] Determine the bounding box of the edge line for each point cloud cluster;

[0035] The bounding box of each point cloud cluster is determined based on the first 2D image;

[0036] For each point cloud cluster, determine the Euclidean distance between the bounding box and the edge bounding box; the point cloud cluster with the smallest Euclidean distance is the target point cloud cluster.

[0037] The present invention also provides a 3D target detection system, comprising:

[0038] The 2D bounding box acquisition module is used to acquire the 2D bounding box of the target to be tested in the first 2D image;

[0039] The second 2D image determination module is used to establish the projection relationship between the first 2D image and the point cloud, determine the 2D pixel corresponding to each point cloud in the 3D point cloud according to the projection relationship, and the 3D information corresponding to the 2D pixel, to obtain a second 2D image containing 3D information, wherein the 3D information includes the depth value of the pixel.

[0040] An extended bounding box determination module is used to determine the extended bounding box in the second 2D image based on the 2D bounding box;

[0041] The 3D information acquisition module is used to acquire the 3D information of all pixels within the extended bounding box;

[0042] The target determination module is used to fit the 3D information of all the pixels to obtain the 3D information of the target.

[0043] Optionally, the target determination module is specifically used for:

[0044] A histogram is determined based on the 3D information of all the pixels, wherein the horizontal axis of the histogram represents the depth value of the pixel, and the vertical axis of the histogram represents the number of pixels.

[0045] Determine the x-axis value corresponding to the peak value of the histogram;

[0046] Obtain 3D information of pixels within a preset range of the horizontal coordinate value; the preset range of the horizontal coordinate value is a preset distance range extended to both sides based on the horizontal coordinate value;

[0047] The 3D information of the target to be tested is determined based on the 3D information of the pixels within the preset range of the horizontal coordinate value.

[0048] The present invention also provides a vehicle including the aforementioned 3D target detection system.

[0049] This invention provides a 3D target detection method, system, and vehicle. The method involves: acquiring the 2D bounding box of the target in a first 2D image; establishing a projection relationship between the first 2D image and a point cloud; determining the 2D pixels corresponding to each point in the 3D point cloud and their corresponding 3D information based on the projection relationship; obtaining a second 2D image containing 3D information, including the depth value of each pixel; determining an extended bounding box in the second 2D image based on the 2D bounding box; acquiring the 3D information of all pixels within the extended bounding box; and fitting the 3D information of all pixels to obtain the 3D information of the target. In essence, this invention establishes a projection relationship between the first 2D image and the point cloud, constructs a second 2D image containing 3D information, and then performs subsequent 3D information detection of the target based on the 3D information of all pixels within the extended bounding box. This achieves the matching and fusion of image and point cloud data. The algorithm is simple, computationally inexpensive, and capable of real-time detection. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0051] Figure 1 This is a flowchart of the 3D target detection method provided by the present invention;

[0052] Figure 2 This is a schematic diagram of the coordinates of the 2D bounding box and the extended bounding box of the target to be measured provided by the present invention;

[0053] Figure 3 This is a block diagram of the 3D target detection system provided by the present invention;

[0054] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0056] The following is combined with Figures 1-4 The present invention describes a 3D target detection method, system, and vehicle. Figure 1 This is a flowchart of the 3D target detection method of the present invention, as shown below. Figure 1 As shown, it includes:

[0057] Step 101: Obtain the 2D bounding box of the target in the first 2D image.

[0058] The first 2D image is a 2D image directly captured by a 2D image acquisition device, such as a 2D camera or webcam.

[0059] In a specific embodiment, such as Figure 2 As shown, in the image coordinate system, the top left corner is the origin (0, 0). Specifically, the 2D bounding box includes the coordinates of the top left corner of the 2D bounding box (umin, vmin), the width and height of the target to be measured (width, height), and the class of the target to be measured in the 2D bounding box.

[0060] In one possible embodiment, obtaining the 2D bounding box of the target in the first 2D image specifically includes:

[0061] Establish a 2D object detection category set 2D-list. The 2D-list includes object categories, and different object categories are denoted as [D1, D2, ...], where D1, D2, ... represent different categories.

[0062] Create a 3D model set (3D-list) in world coordinates corresponding to the category set (2D-list). The 3D-list contains the width and height information of different target categories, denoted as [D1,h1,w1], [D2,h2,w2], ..., where h1, h2, ... represent different heights, and w1, w2, ... represent different widths.

[0063] Image detection algorithms are used to detect 2D-list targets in images. This involves obtaining the category of the target and its width and height information, and then using this width and height information to obtain the 2D bounding box information of the target in the image pixels. This yields the coordinates (umin, vmin) of the top-left corner of the target's 2D bounding box, the target's width and height, and the target's category (class) within the 2D bounding box.

[0064] Step 102: Establish the projection relationship between the first 2D image and the point cloud, determine the 2D pixel corresponding to each point cloud in the 3D point cloud and the 3D information corresponding to the 2D pixel according to the projection relationship, and obtain a second 2D image containing 3D information, wherein the 3D information includes the depth value of the pixel.

[0065] Among them, 3D point clouds are collected by 3D point cloud acquisition devices, such as LiDAR.

[0066] In one specific embodiment, step 102 specifically includes:

[0067] The intrinsic and extrinsic parameters of the 3D point cloud projected onto the first 2D image are determined by calibration.

[0068] A storage container for 3D information is constructed. The storage container is a collection of storage spaces for each pixel, indexed by the pixels of the first 2D image, for storing the 3D information of the pixels.

[0069] The 2D pixel points corresponding to each point cloud in the 3D point cloud are determined based on the intrinsic and extrinsic parameters.

[0070] The 3D information corresponding to the 2D pixel is determined based on the storage container to obtain a second 2D image containing the 3D information.

[0071] The first 2D image is a 2D image that does not include 3D information, and the second 2D image is a 2D image that includes 3D information.

[0072] In one possible embodiment, the intrinsic and extrinsic parameters include the camera's intrinsic parameters, distortion parameters, and the rotation and translation matrices of the point cloud projected onto the image.

[0073] In one possible embodiment, constructing a 3D information storage container specifically includes:

[0074] Using the pixel coordinates of the first 2D image as an index, a storage point is established for each pixel position, and the entire image corresponds to a container, thus establishing a 3D point storage container in the image coordinate system. By obtaining the intrinsic and extrinsic parameters, the coordinates of the 2D points in the image coordinate system corresponding to each 3D point cloud are calculated, and the 3D information contained in each 3D point cloud is stored in the corresponding 3D point storage container in the image coordinate system, thus obtaining a 3D information storage container.

[0075] The 3D information includes the 3D coordinates of the pixel.

[0076] Step 103: Determine the expanded bounding box in the second 2D image based on the 2D bounding box.

[0077] In one specific embodiment, step 103 includes:

[0078] Obtain the coordinates of the center point of the 2D bounding box;

[0079] In the second 2D image, the expanded bounding box is obtained by expanding the 2D bounding box by a preset ratio in both the width and height directions according to the coordinates of the center point.

[0080] In one specific embodiment, the expanded bounding box is generated by expanding the 2D bounding box by a preset ratio in both width and height directions at the coordinates of the 2D bounding box center point. The purpose of this step is to include as much point cloud data as possible within the bounding box to improve the accuracy of distance calculation.

[0081] In one possible embodiment, the preset ratio is 1 / 4, resulting in an expanded bounding box with an aspect ratio of 3 / 2 of the 2D bounding box. The coordinates of the top-left corner of the expanded bounding box are (umin - 1 / 4 width, vmin - 1 / 4 height), as shown below. Figure 2 As shown, the width and height are (3 / 2 width, 3 / 2 height).

[0082] Step 104: Obtain the 3D information of all pixels within the extended bounding box.

[0083] Step 105: Fit the 3D information of all the pixels to obtain the 3D information of the target to be tested.

[0084] In one specific embodiment, step 105 specifically includes:

[0085] Step 1051: Determine a histogram based on the 3D information of all pixels, wherein the horizontal axis of the histogram is the depth value of the pixel, and the vertical axis of the histogram is the number of pixels.

[0086] In one specific embodiment, all 3D point information in the 3D point storage container is extracted from the extended bounding box of the target under test. n distance intervals are established, constrained by the distance interval Dinterval, the maximum distance Dmax, and the minimum distance Dmin, where n = (Dmax - Dmin) / Dinterval, and the range of each distance interval is [Dmin + (n-1) * Dinterval, Dmin + n * Dinterval]. The number of pixels whose depth values ​​fall within the above distance intervals is counted to form a distance histogram. Here, the distance interval represents the range of pixel depth values. For example, if the maximum pixel distance is 3000mm and the minimum pixel distance is 500mm, and Dinterval is 500mm, then n = 5; if the number of pixels with depth values ​​in the range of 1000mm-15000mm is 20, then the vertical axis of the histogram is 20.

[0087] The maximum distance is the maximum value of the pixel depth value among all 3D point information, and the minimum distance is the minimum value of the pixel depth value among all 3D point information. The distance interval is set according to the difference between the maximum distance and the minimum distance. Generally, it is necessary to ensure about 10 distance intervals. For example, the difference between the maximum distance Dmax and the minimum distance Dmin is 1 meter. The distance interval Dinterval can be set to 0.1, and n is 10.

[0088] Step 1052: Determine the x-axis value corresponding to the peak value of the histogram.

[0089] In one specific embodiment, step 1052 specifically includes:

[0090] Determine the number of peaks m in the histogram; when the number of peaks m equals 1, determine the x-coordinate value corresponding to the peak value of the histogram; when the number of peaks m is greater than 1, determine the x-coordinate value corresponding to each peak value of the histogram.

[0091] Step 1053: Obtain 3D information of pixels within the preset range of the horizontal coordinate value. The preset range of the horizontal coordinate value is a preset distance range extended to both sides based on the horizontal coordinate value.

[0092] Step 1054: Determine the 3D information of the target to be tested based on the 3D information of the pixels within the preset range of the horizontal coordinate value.

[0093] In a specific embodiment, steps 1053-1054 specifically include:

[0094] When the number of peaks in the histogram is m=1, determine the horizontal coordinate value corresponding to the peak value, and take the information of all 3D points within the preset range of the horizontal coordinate value.

[0095] Based on all 3D point information, determine the minimum depth value min3D, the maximum depth value max3D, and the median depth value center3D of each pixel.

[0096] Fitting min3D, max3D, and center3D yields the 3D information of the target object.

[0097] The "centroid calculation algorithm" is used to fit min3D, max3D, and center3D to output the 3D information of the target. This 3D information includes the depth value of the pixel, the width and height of the target, and the category of the target.

[0098] Furthermore, the preset range for the horizontal coordinate value can be ±0.4 meters.

[0099] Specifically, when the number of peaks m in the histogram is greater than 1, the 3D information of the target to be measured is determined based on the 3D information of the pixels within a preset range of the horizontal coordinate values, including:

[0100] Determine the x-axis value corresponding to each peak of the histogram;

[0101] Define the point cloud corresponding to all pixels within a preset range of each horizontal coordinate value as a point cloud cluster, and obtain the 3D information of each point cloud cluster;

[0102] The target point cloud cluster is determined based on the 3D information of each point cloud cluster;

[0103] The target point cloud cluster is fitted to obtain the 3D information of the target under test.

[0104] Furthermore, determining the target point cloud cluster based on the 3D information of each point cloud cluster includes:

[0105] Determine the bounding box of the edge line for each point cloud cluster;

[0106] The bounding box of each point cloud cluster is determined based on the first 2D image;

[0107] For each point cloud cluster, determine the Euclidean distance between the bounding box and the edge bounding box; the point cloud cluster with the smallest Euclidean distance is the target point cloud cluster.

[0108] In a specific embodiment, steps 1053-1054 specifically include:

[0109] When the number of peaks m in the histogram is greater than 1, the horizontal coordinate value corresponding to each peak of the histogram is determined, and the 3D point cloud corresponding to all pixels within a preset range of each horizontal coordinate value is defined as a point cloud cluster.

[0110] Calculate the edge points of each point cloud cluster, where the edge points are the locations in the point cloud where the depth value of a pixel changes abruptly.

[0111] Based on the edge points, edge lines are generated through linear fitting. Each point cloud cluster generates 4 edge lines, namely in the up, down, left, and right directions, to obtain the bounding box of the edge lines of each point cloud cluster, and calculate the 3D width and height information of the bounding box of the edge lines.

[0112] Based on the target category generated by 2D detection, find the width and height information of the bounding box in the corresponding 3D-list.

[0113] Compare the Euclidean distance D between the 3D width and height information of the bounding box and the width and height information of the bounding box in the 3D-list, where D = sqrt((w-w')*(w-w')+(h-h')*(h-h')), where w is the width of the bounding box, w' is the width of the bounding box, h is the height of the bounding box, and h' is the height of the bounding box.

[0114] The point cloud cluster with the smallest Euclidean distance D is the 3D point cloud cluster of the target to be measured, which is the target point cloud cluster.

[0115] Based on the 3D point information of the 3D point cloud cluster of the target to be tested, determine the minimum depth value min3D, the maximum depth value max3D, and the median depth value center3D of the pixel.

[0116] Fitting min3D, max3D, and center3D yields the 3D information of the target object.

[0117] The 3D target detection system provided by the present invention is described below. The 3D target detection system described below can be referred to in correspondence with the 3D target detection method described above.

[0118] Figure 3 The 3D target detection system provided by this invention, such as Figure 3 As shown, a 3D target detection system includes:

[0119] The 2D bounding box acquisition module 301 is used to acquire the 2D bounding box of the target to be tested in the first 2D image.

[0120] The second 2D image determination module 302 is used to establish the projection relationship between the first 2D image and the point cloud, determine the 2D pixel points corresponding to each point cloud in the 3D point cloud according to the projection relationship, and the 3D information corresponding to the 2D pixel points, to obtain a second 2D image containing 3D information, wherein the 3D information includes the depth value of the pixel points.

[0121] The extended bounding box determination module 303 is used to determine the extended bounding box in the second 2D image based on the 2D bounding box.

[0122] The 3D information acquisition module 304 is used to acquire the 3D information of all pixels within the extended bounding box.

[0123] The target determination module 305 is used to fit the 3D information of all the pixels to obtain the 3D information of the target.

[0124] In one specific embodiment, the target determination module 305 is specifically used for:

[0125] A histogram is determined based on the 3D information of all the pixels, wherein the horizontal axis of the histogram represents the depth value of the pixel, and the vertical axis of the histogram represents the number of pixels.

[0126] Determine the x-axis value corresponding to the peak value of the histogram;

[0127] Obtain 3D information of pixels within a preset range of the horizontal coordinate value; the preset range of the horizontal coordinate value is a preset distance range extended to both sides based on the horizontal coordinate value;

[0128] The 3D information of the target to be tested is determined based on the 3D information of the pixels within the preset range of the horizontal coordinate value.

[0129] In one specific embodiment, the second 2D image determination module 302 is specifically used for:

[0130] The intrinsic and extrinsic parameters of the 3D point cloud projected onto the first 2D image are determined by calibration;

[0131] A storage container for 3D information is constructed. The storage container is a set of storage spaces for each pixel, indexed by the pixels of the first 2D image, for storing the 3D information of the pixels.

[0132] The 2D pixel points corresponding to each point cloud in the 3D point cloud are determined based on the intrinsic and extrinsic parameters.

[0133] The 3D information corresponding to the 2D pixel is determined based on the storage container to obtain a second 2D image containing the 3D information.

[0134] In one specific embodiment, the extended bounding box determination module 303 is specifically used for:

[0135] Obtain the coordinates of the center point of the 2D bounding box;

[0136] In the second 2D image, the expanded bounding box is obtained by expanding the 2D bounding box by a preset ratio in both the width and height directions according to the coordinates of the center point.

[0137] In one specific embodiment, determining the abscissa value corresponding to the peak value of the histogram specifically includes:

[0138] Determine the number of peaks in the histogram; when the number of peaks in the histogram is equal to 1, determine the x-coordinate value corresponding to the peak value of the histogram; when the number of peaks in the histogram is greater than 1, determine the x-coordinate value corresponding to each peak value of the histogram.

[0139] In one specific embodiment, when the number of peaks in the histogram is greater than 1, the 3D information of the target to be measured is determined based on the 3D information of pixels within a preset range of the horizontal coordinate values, including:

[0140] Determine the x-axis value corresponding to each peak of the histogram;

[0141] Define the point cloud corresponding to all pixels within a preset range of each horizontal coordinate value as a point cloud cluster, and obtain the 3D information of each point cloud cluster;

[0142] The target point cloud cluster is determined based on the 3D information of each point cloud cluster;

[0143] The target point cloud cluster is fitted to obtain the 3D information of the target under test.

[0144] In one specific embodiment, determining the target point cloud cluster based on the 3D information of each point cloud cluster includes:

[0145] Determine the bounding box of the edge line for each point cloud cluster;

[0146] The bounding box of each point cloud cluster is determined based on the first 2D image;

[0147] For each point cloud cluster, determine the Euclidean distance between the bounding box and the edge bounding box; the point cloud cluster with the smallest Euclidean distance is the target point cloud cluster.

[0148] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a 3D target detection method, which includes:

[0149] Obtain the 2D bounding box of the target object in the first 2D image.

[0150] Establish the projection relationship between the first 2D image and the point cloud, determine the 2D pixel corresponding to each point cloud in the 3D point cloud and the 3D information corresponding to the 2D pixel according to the projection relationship, and obtain a second 2D image containing 3D information, wherein the 3D information includes the depth value of the pixel.

[0151] The expanded bounding box in the second 2D image is determined based on the 2D bounding box.

[0152] Obtain 3D information of all pixels within the extended bounding box.

[0153] The 3D information of all the pixels is fitted to obtain the 3D information of the target under test.

[0154] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing a 3D target detection method, the method comprising:

[0156] Obtain the 2D bounding box of the target object in the first 2D image.

[0157] Establish the projection relationship between the first 2D image and the point cloud, determine the 2D pixel corresponding to each point cloud in the 3D point cloud and the 3D information corresponding to the 2D pixel according to the projection relationship, and obtain a second 2D image containing 3D information, wherein the 3D information includes the depth value of the pixel.

[0158] The expanded bounding box in the second 2D image is determined based on the 2D bounding box.

[0159] Obtain 3D information of all pixels within the extended bounding box.

[0160] The 3D information of all the pixels is fitted to obtain the 3D information of the target under test.

[0161] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform a 3D target detection method, the method comprising:

[0162] Obtain the 2D bounding box of the target object in the first 2D image.

[0163] Establish the projection relationship between the first 2D image and the point cloud, determine the 2D pixel corresponding to each point cloud in the 3D point cloud and the 3D information corresponding to the 2D pixel according to the projection relationship, and obtain a second 2D image containing 3D information, wherein the 3D information includes the depth value of the pixel.

[0164] The expanded bounding box in the second 2D image is determined based on the 2D bounding box.

[0165] Obtain 3D information of all pixels within the extended bounding box.

[0166] The 3D information of all the pixels is fitted to obtain the 3D information of the target under test.

[0167] In another aspect, the present invention also provides a vehicle, including the aforementioned 3D target detection system or the aforementioned electronic device.

[0168] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0169] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 3D target detection method, characterized in that, include: Obtain the 2D bounding box of the target in the first 2D image; Establish the projection relationship between the first 2D image and the point cloud, determine the 2D pixel corresponding to each point cloud in the 3D point cloud and the 3D information corresponding to the 2D pixel according to the projection relationship, and obtain a second 2D image containing 3D information, wherein the 3D information includes the depth value of the pixel; the first 2D image is a 2D image that does not include 3D information, and the second 2D image is a 2D image that includes 3D information. Determine the expanded bounding box in the second 2D image based on the 2D bounding box; Obtain 3D information of all pixels within the extended bounding box; The 3D information of all the pixels is fitted to obtain the 3D information of the target under test; The step of establishing the projection relationship between the first 2D image and the point cloud, determining the 2D pixels corresponding to each point cloud in the 3D point cloud based on the projection relationship, and the 3D information corresponding to the 2D pixels, to obtain a second 2D image containing 3D information, includes: The intrinsic and extrinsic parameters of the 3D point cloud projected onto the first 2D image are determined by calibration; A storage container for 3D information is constructed. The storage container is a set of storage spaces for each pixel, indexed by the pixels of the first 2D image, for storing the 3D information of the pixels. The 2D pixel points corresponding to each point cloud in the 3D point cloud are determined based on the intrinsic and extrinsic parameters. The 3D information corresponding to the 2D pixel is determined based on the storage container to obtain a second 2D image containing the 3D information.

2. The 3D target detection method according to claim 1, characterized in that, The process of fitting the 3D information of all pixels to obtain the 3D information of the target to be tested includes: A histogram is determined based on the 3D information of all the pixels, wherein the horizontal axis of the histogram is the depth value of the pixel, and the vertical axis of the histogram is the number of pixels. Determine the x-axis value corresponding to the peak value of the histogram; Obtain 3D information of pixels within a preset range of the horizontal coordinate value; the preset range of the horizontal coordinate value is a preset distance range extended to both sides based on the horizontal coordinate value; The 3D information of the target to be tested is determined based on the 3D information of the pixels within the preset range of the horizontal coordinate value.

3. The 3D target detection method according to claim 1 or 2, characterized in that, Determining the expanded bounding box in the second 2D image based on the 2D bounding box includes: Obtain the coordinates of the center point of the 2D bounding box; In the second 2D image, the expanded bounding box is obtained by expanding the 2D bounding box by a preset ratio in both the width and height directions according to the coordinates of the center point.

4. The 3D target detection method according to claim 2, characterized in that, Determining the x-coordinate value corresponding to the peak value of the histogram includes: Determine the number of peaks in the histogram; when the number of peaks in the histogram is equal to 1, determine the x-coordinate value corresponding to the peak value of the histogram; when the number of peaks in the histogram is greater than 1, determine the x-coordinate value corresponding to each peak value of the histogram.

5. The 3D target detection method according to claim 4, characterized in that, When the number of peaks in the histogram is greater than 1, the 3D information of the target to be measured is determined based on the 3D information of pixels within a preset range of the horizontal coordinate values, including: Determine the x-axis value corresponding to each peak of the histogram; Define the point cloud corresponding to all pixels within a preset range of each horizontal coordinate value as a point cloud cluster, and obtain the 3D information of each point cloud cluster; The target point cloud cluster is determined based on the 3D information of each point cloud cluster; The target point cloud cluster is fitted to obtain the 3D information of the target under test.

6. The 3D target detection method according to claim 5, characterized in that, The step of determining the target point cloud cluster based on the 3D information of each point cloud cluster includes: Determine the bounding box of the edge line for each point cloud cluster; The bounding box of each point cloud cluster is determined based on the first 2D image; For each point cloud cluster, determine the Euclidean distance between the bounding box and the edge bounding box; the point cloud cluster with the smallest Euclidean distance is the target point cloud cluster.

7. A 3D target detection system, characterized in that, include: The 2D bounding box acquisition module is used to acquire the 2D bounding box of the target to be tested in the first 2D image; The second 2D image determination module is used to establish the projection relationship between the first 2D image and the point cloud, determine the 2D pixel points corresponding to each point cloud in the 3D point cloud according to the projection relationship, and the 3D information corresponding to the 2D pixel points, to obtain a second 2D image containing 3D information, wherein the 3D information includes the depth value of the pixel points; the first 2D image is a 2D image that does not include 3D information, and the second 2D image is a 2D image that includes 3D information. An extended bounding box determination module is used to determine the extended bounding box in the second 2D image based on the 2D bounding box; The 3D information acquisition module is used to acquire the 3D information of all pixels within the extended bounding box; The target determination module is used to fit the 3D information of all the pixels to obtain the 3D information of the target. The step of establishing the projection relationship between the first 2D image and the point cloud, determining the 2D pixels corresponding to each point cloud in the 3D point cloud based on the projection relationship, and the 3D information corresponding to the 2D pixels, to obtain a second 2D image containing 3D information, includes: The intrinsic and extrinsic parameters of the 3D point cloud projected onto the first 2D image are determined by calibration; A storage container for 3D information is constructed. The storage container is a set of storage spaces for each pixel, indexed by the pixels of the first 2D image, for storing the 3D information of the pixels. The 2D pixel points corresponding to each point cloud in the 3D point cloud are determined based on the intrinsic and extrinsic parameters. The 3D information corresponding to the 2D pixel is determined based on the storage container to obtain a second 2D image containing the 3D information.

8. The 3D target detection system according to claim 7, characterized in that, The target determination module is specifically used for: A histogram is determined based on the 3D information of all the pixels, wherein the horizontal axis of the histogram represents the depth value of the pixel, and the vertical axis of the histogram represents the number of pixels. Determine the x-axis value corresponding to the peak value of the histogram; Obtain 3D information of pixels within a preset range of the horizontal coordinate value; the preset range of the horizontal coordinate value is a preset distance range extended to both sides based on the horizontal coordinate value; The 3D information of the target to be tested is determined based on the 3D information of the pixels within the preset range of the horizontal coordinate value.

9. A vehicle, characterized in that, Including the 3D target detection system as described in claim 7 or 8.

Images