3D target detection method, computer program product and terminal

Abstract

The invention discloses a 3D target detection method, a computer program product and a terminal, belongs to the technical field of image processing, and jointly calibrates a laser radar and an image acquisition device. Mapping the global feature of the point cloud to an image feature map by using a transformation kernel, constructing a sparse matrix, and flattening the image feature map into a matrix; performing batch processing and normalization processing on the sparse matrix and the matrix, and then performing connection processing to obtain fusion features of the image and the point cloud; performing region proposal by using the fusion features to generate candidate target regions; and carrying out classification and bounding box regression on the candidate regions, determining the category and the position of the target, and obtaining a 3D target detection result. The mapping relation between the point cloud and the monocular image is constructed by using the sparse point cloud, and it is ensured that only image pixel point coordinates and point cloud coordinate pairs sharing the same point can be paired, so that accurate alignment of features is realized, the accuracy of target detection is improved, and the precision of 3D target detection is ensured.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a 3D target detection method, a computer program product and a terminal that fuses point cloud and image. Background Art

[0002] 3D object detection is a technology that uses sensor data (such as images collected by cameras and point cloud data collected by lidar) to identify and locate targets in three-dimensional space. It provides key information for autonomous driving, robot navigation, etc. by predicting the position, size and direction of the target. In order to obtain more comprehensive and accurate environmental information, it is usually necessary to fuse data from different sensors. It has become a vital task to make full use of the advantages of various modal information and design safe and efficient environmental perception technology. In particular, the exploration of multimodal fusion detection technology will be a key direction in the development of future autonomous driving systems. Multimodal perception not only improves perception accuracy, but also provides more accurate and rich information for subsequent path planning and decision-making. For example, the distance data provided by lidar combined with the object recognition results of the camera can help the system better predict the intentions of other vehicles or pedestrians, thereby optimizing driving decisions and vehicle control. However, images and point cloud data come from different sensors, which have different characteristics and limitations when capturing environmental information. Images provide rich color and texture information, but lack depth information; while lidar provides accurate three-dimensional spatial information, but is limited in resolution and robustness to ambient light changes. It can be seen that point cloud data and images have different representation methods and originate from different coordinate systems, which leads to the problem of misalignment between radar point cloud and image features during the fusion process of point cloud data and images, which in turn leads to inaccurate target positioning, thus affecting the accuracy of 3D target detection. Summary of the invention

[0003] The purpose of the present invention is to overcome the problems of the prior art and provide a 3D object detection method, a computer program product and a terminal.

[0004] The object of the present invention is to achieve the following technical solution: a 3D target detection method, the method comprising the following steps:

[0005] Joint calibration of radar and image acquisition devices;

[0006] Perform feature extraction and feature aggregation on point cloud data to obtain the global features of the point cloud;

[0007] Perform feature extraction and sparse non-uniform pooling on the image to obtain an image feature map;

[0008] Use the transformation kernel to map the global features of the point cloud to the image feature map, construct a sparse matrix, and flatten the image feature map into a matrix; perform batch normalization on the sparse matrix and the matrix, and then perform concatenation to obtain the fusion features of the image and the point cloud;

[0009] Use fusion features to propose regions and generate candidate target regions;

[0010] The candidate regions are classified and bounding box regressed to determine the category and location of the target and obtain the 3D target detection result.

[0011] In one example, a point cloud segmentation neural network PointNet is used to perform feature extraction and feature aggregation processing on point cloud data.

[0012] In one example, feature extraction and feature aggregation processing are performed on point cloud data, including:

[0013] Use PointNet to process point cloud data and generate feature tensors;

[0014] For each feature dimension of the feature tensor, a maximum pooling operation is performed on all points to obtain a global feature vector.

[0015] In one example, before performing feature extraction and feature aggregation processing on point cloud data, the following steps are also included:

[0016] Discretize the point cloud data into multiple fixed-size columns on a two-dimensional plane;

[0017] Calculate the distance to the arithmetic mean of all points in each bin and the offset of each point to the bin center, and represent each point as a nine-dimensional feature vector.

[0018] In one example, after extracting features from an image and before sparse non-uniform pooling processing, the method further includes:

[0019] Target features at different scales are extracted through multiple levels of the convolutional network to achieve multi-scale feature fusion processing.

[0020] In one example, the calculation expression for transform kernel sparsification is:

[0021] k x,y (u,v)=δ (x,y)(xi,yi) k x,y (u,v)δ (u,v)(uj,vj) ,i,j=1,2…N

[0022] Among them, k x,y(u,v) represents the transformation kernel; x, y are the horizontal and vertical coordinates of the point cloud respectively; u, v are the horizontal and vertical coordinates of the pixel points in the image respectively; δ represents the Dirac function, which is used as an indicator function in sparse mapping; i represents the point cloud label; j represents the pixel label in the image; N represents the upper limit of the label; δ (x,y)(xi,yi) Used to indicate whether a point (x, t) in the point cloud coincides with a point (xi, yi) in the point cloud; δ (u,v)(uj,vj) Indicates whether the pixel coordinates (u, v) coincide with the pixel coordinates (uj, vj) after point cloud mapping.

[0023] It should be further explained that the technical features corresponding to the above examples can be combined or replaced with each other to form a new technical solution.

[0024] The present invention also includes a computer program product, including a computer program, which, when executed by a processor, implements the steps of the 3D object detection method formed by any one of the above examples or a combination of multiple examples.

[0025] The present invention also includes a storage medium on which computer instructions are stored. When the computer instructions are executed, the steps of the 3D object detection method formed by any one or more of the above examples are executed.

[0026] The present invention also includes a terminal, including a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, and when the processor executes the computer instructions, the steps of the 3D target detection method formed by any one or more of the above examples are executed.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. In one example, the joint calibration of the radar and image acquisition device can fix the posture relationship between the two, and then use the sparse point cloud to construct the mapping relationship between the point cloud and the monocular image, ensuring that only the image pixel coordinates and point cloud coordinate pairs that share the same points will be paired, thereby achieving accurate alignment of features and improving the accuracy of target detection, thereby ensuring the accuracy of 3D target detection; the sparse non-uniform pooling layer realizes the effective conversion and fusion of point cloud and image features through sparse matrix multiplication. Only when the corresponding point cloud data exists, the pixels in the image will be used for fusion, avoiding unnecessary calculations and potential information loss, and further ensuring the accuracy of target positioning.

[0029] At the same time, point cloud data provides accurate 3D geometric information of objects, while monocular cameras provide rich visual information, such as texture, color, etc. By fusing these two types of data, it is possible to combine geometric information and visual information, making 3D target detection more accurate, especially in complex scenes such as occlusion and low brightness.

[0030] 2. In one example, by performing columnar division and feature enhancement processing on point cloud data, high-dimensional point cloud data can be effectively processed and fused, and objects in 3D space can be better identified, thereby improving detection accuracy.

[0031] 3. In one example, by extracting target features at different scales, the ability to perceive the target shape and size can be enhanced, further improving the accuracy of 3D target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The specific implementation methods of the present invention are further described in detail below in conjunction with the accompanying drawings. The accompanying drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The same reference numerals are used in these drawings to represent the same or similar parts. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0033] Figure 1 A method flow chart provided for an example of the present invention;

[0034] Figure 2 A method framework diagram provided for an example of the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] In the description of the present invention, it should be noted that the directions or positional relationships indicated by "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. are directions or positional relationships based on the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the use of ordinal numbers (e.g., "first and second", "first to fourth", etc.) is to distinguish objects, and is not limited to this order, and cannot be understood as indicating or implying relative importance.

[0037] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0038] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0039] A single sensor is easily limited by its own physical properties, such as the sparse data collection of point clouds for distant objects, and the lack of depth information in monocular cameras. Fusion of point cloud and monocular camera data can complement each other's advantages, improve adaptability to different environments and scenes, and make detection more robust and reliable. Therefore, the present invention proposes a 3D object detection method that fuses point clouds and images.

[0040] In one example, if Figure 1-Figure 2 As shown, a 3D target detection method for fusing point cloud and image, the method comprises the following steps:

[0041] S10: Jointly calibrate the radar and image acquisition device.

[0042] Among them, the image acquisition device is a camera, a webcam, etc. For the calibration of the camera, first download the template file for camera calibration: the calibration plate PDF file, print it, and use the checkerboard calibration method to obtain the intrinsic parameters of the camera. Then calibrate the lidar and the camera together to fix the posture relationship between the camera and the radar. First, the camera uses the checkerboard calibration method to obtain the camera's intrinsic parameter matrix, and then the camera and lidar are jointly calibrated to obtain the external parameters, that is, the rotation matrix and translation vector of the lidar relative to the camera. The internal parameters are mainly used for image dedistortion and pixel to camera coordinate conversion. The external parameters are used to achieve spatial alignment between different sensor coordinate systems to ensure accurate projection and fusion of image and point cloud data under BEV.

[0043] S20: Perform feature extraction and feature aggregation processing on the point cloud data to obtain the global features of the point cloud.

[0044] Through the point cloud segmentation network, shallow features are first extracted from the original point cloud data, and then the global and local feature information is extracted and fused to obtain the deep features of the point cloud.

[0045] S30: performing feature extraction and sparse non-uniform pooling processing on the image to obtain an image feature map.

[0046] The feature extraction network is used to extract the visual feature information of the monocular image, and then the sparse non-uniform pooling layer is used for sparse non-uniform pooling. The sparse non-uniform pooling layer realizes the effective conversion and fusion of features by sparse matrix multiplication. The sparse non-uniform pooling layer flattens the front view (monocular view) feature map into a dense matrix, and obtains the pooled feature map by multiplying it with the sparse matrix. This means that pixels in the image will only be used for fusion when the corresponding point cloud data exists, avoiding unnecessary calculations and potential information loss. Due to the sparsity of the point cloud, this calculation is efficient and can maintain the correspondence between features, which can ensure the accuracy of 3D target detection.

[0047] S40: Use the transformation kernel to map the global features of the point cloud to the image feature map, construct a sparse matrix, and flatten the image feature map into a matrix; perform batch normalization on the sparse matrix and the matrix, and then perform concatenation processing to obtain the fusion features of the image and the point cloud.

[0048] The mapping relationship between the point cloud and the monocular image is constructed using sparse point cloud, ensuring that only the image pixel coordinates and point cloud coordinate pairs that share the same points are paired, thereby achieving accurate alignment of features and improving the accuracy of target detection, thereby ensuring the accuracy of 3D target detection.

[0049] At the same time, this application performs data fusion of point cloud and monocular image before the region proposal stage, which means that the entire feature map is used for fusion processing, not just the region of interest, thereby providing richer contextual information. This method can more effectively utilize multi-view features, reduce the problem of feature misalignment, and improve the performance and robustness of 3D object detection.

[0050] S50: Perform region proposal using the fused features to generate candidate target regions. Specifically, a region proposal network (such as RPN) is used to generate candidate regions that may contain targets.

[0051] S60: Based on the detection head, the candidate area is classified and the bounding box is regressed to determine the category and position of the target, and a 3D target detection result is obtained.

[0052] Preferably, after the bounding box regression process, overlapping or redundant detection results are removed by techniques such as non-maximum suppression (NMS) to obtain the final 3D object detection result.

[0053] In one example, a point cloud segmentation neural network PointNet is used to perform feature extraction and feature aggregation processing on point cloud data. Specifically, the point cloud segmentation neural network PointNet is used to process the points in each column to generate a feature tensor of shape [N, D], where N is the number of points in the point cloud and D is the feature dimension extracted from each point; the feature channel is max-operated, and each feature dimension of the feature tensor is max-pooled on all points to obtain a global feature vector of length D.

[0054] In one example, before performing feature extraction and feature aggregation processing on point cloud data, the following steps are also included:

[0055] Columnar division: discretize the point cloud data into multiple fixed-size pillars on a two-dimensional plane (xy plane). Each pillar is a specific two-dimensional grid unit, which is set according to the laser radar beam and the number of point clouds in the Pillar.

[0056] Feature enhancement: Calculate the distance c of the arithmetic mean of all points in each column and the offset p of each point to the center of the column; for each point in each column, enhance its feature information and use x c ,y c ,z c ,x p and p Increase the points in each bin and represent each point as a 9-dimensional feature vector.

[0057] In one example, feature extraction is performed on a monocular image through a feature extraction network such as a convolutional neural network to obtain rich visual features. Preferably, after feature extraction is performed on the image and before sparse non-uniform pooling processing, the method further includes:

[0058] Target features at different scales are extracted through multiple levels of the convolutional network to achieve multi-scale feature fusion processing, ensuring that target objects of various sizes in the image are detected.

[0059] In one example, the monocular image view input size is H f W f , the size of the bird's-eye view (the fusion image of the monocular image and the point cloud data) is L b ×W b , the radar point cloud size is {(x i ,y i , z i )|i=1,2,…N}, then the calculation expression of transform kernel sparsification is:

[0060] k x,y (u,v)=δ (x,y)(xi,yi) k x,y (u,v)δ(u,v)(uj,vj) ,i,j=1,2…N

[0061] Among them, k x,y (u,v) represents the transformation kernel; x, y are the horizontal and vertical coordinates of the point cloud respectively; u, v are the horizontal and vertical coordinates of the pixel points in the image respectively; δ represents the Dirac function, which is used as an indicator function in sparse mapping; i represents the point cloud label; j represents the pixel label in the image; N represents the upper limit of the label; δ (x,y)(xi,yi) Used to indicate whether a point (x, y) in the point cloud coincides with a point (xi, yi) in the point cloud; δ (u,v)(uj,vj) Indicates whether the pixel coordinates (u, v) coincide with the pixel coordinates (uj, vj) after point cloud mapping. At this time, the feature map (such as an image) is used as input, and (u i ,v i )、(x i ,y i ) as a parameter, constructing an L b W b ×H f W f The sparse matrix M of H f ×W f ×C feature maps are flattened into dense H f W f ×C matrix B = MF; then the sparse matrix M and matrix B are batch normalized and connected to obtain the fusion features of the image and point cloud.

[0062] In this example, the fusion processing of image and point cloud is realized. The 2D image captured by the monocular camera provides detailed information of the object for 3D detection, which helps to more accurately identify and classify objects. Point cloud data can accurately estimate the shape and size of the object. Combined with the camera data, it can improve the classification accuracy of target detection. Furthermore, point cloud data becomes sparse when detecting long-distance targets, while the monocular camera can maintain good long-distance resolution. By fusing the data of the two, the deficiency of point cloud data can be compensated in long-distance target detection, ensuring the detection effect in long-distance scenes. In addition, compared with the use of high-precision LiDAR or binocular camera systems, the calculation and deployment cost of monocular cameras is lower, and 2D image data can be processed quickly. Through reasonable multimodal fusion technology, unnecessary redundant calculations can be reduced, the calculation efficiency can be improved, and the real-time requirements of 3D target detection can be met.

[0063] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the 3D object detection method formed by any one of the above examples or a combination of multiple examples. The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0064] The present invention also provides a storage medium, which has the same inventive concept as a 3D target detection method formed by any one of the above examples or a combination of multiple examples, and stores computer instructions on the storage medium, which execute the steps of the 3D target detection method formed by any one of the above examples or a combination of multiple examples when executed.

[0065] Based on this understanding, the technical solution of this embodiment, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0066] The present invention also provides a terminal, which has the same inventive concept as any example or a combination of multiple examples corresponding to the above-mentioned 3D object detection method, including a memory and a processor, the memory stores computer instructions that can be run on the processor, and the processor executes the steps of the above-mentioned 3D object detection method when running the computer instructions. The processor can be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0067] In one example, the terminal, i.e., the electronic device, is presented in the form of a general-purpose computing device, and the components of the electronic device may include but are not limited to: at least one processing unit (processor) mentioned above, at least one storage unit mentioned above, and a bus connecting different system components (including storage units and processing units).

[0068] The storage unit stores a program code, and the program code can be executed by the processing unit, so that the processing unit performs the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification. For example, the processing unit can perform the above-mentioned 3D object detection method.

[0069] The storage unit may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 3201 and / or a cache memory unit, and may further include a read-only memory unit (ROM).

[0070] The storage unit may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0071] The bus may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0072] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0073] Through the above description, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method of the exemplary embodiment of the present application.

[0074] The above specific implementation methods are detailed descriptions of the present invention. It cannot be determined that the specific implementation methods of the present invention are limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions and substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A 3D object detection method, characterized in that: The following steps are involved: Joint calibration of radar and image acquisition devices; Perform feature extraction and feature aggregation on point cloud data to obtain the global features of the point cloud; Perform feature extraction and sparse non-uniform pooling on the image to obtain an image feature map; Use the transformation kernel to map the global features of the point cloud to the image feature map, construct a sparse matrix, and flatten the image feature map into a matrix; The sparse matrix and matrix are batch normalized and then connected to obtain the fusion features of the image and point cloud; Use fusion features to propose regions and generate candidate target regions; The candidate regions are classified and bounding box regressed to determine the category and location of the target and obtain the 3D target detection result.

2. The 3D target detection method according to claim 1, characterized in that: The point cloud segmentation neural network PointNet is used to perform feature extraction and feature aggregation on point cloud data.

3. The 3D target detection method according to claim 2, characterized in that: Perform feature extraction and feature aggregation processing on point cloud data, including: Use PointNet to process point cloud data and generate feature tensors; For each feature dimension of the feature tensor, a maximum pooling operation is performed on all points to obtain a global feature vector.

4. The 3D object detection method according to claim 1, characterized in that: Before feature extraction and feature aggregation of point cloud data, it also includes: Discretize the point cloud data into multiple fixed-size columns on a two-dimensional plane; Calculate the distance to the arithmetic mean of all points in each bin and the offset of each point to the bin center, and represent each point as a nine-dimensional feature vector.

5. The 3D object detection method according to claim 1, characterized in that: After feature extraction of the image and before sparse non-uniform pooling, it also includes: Target features at different scales are extracted through multiple levels of the convolutional network to achieve multi-scale feature fusion processing.

6. The 3D object detection method according to claim 1, characterized in that: The calculation expression of transform kernel sparsification is: k x,y (u,v)=δ (x,y)(xi,yi) k x,y (u,v)δ (u,v)(uj,vj) ,i,j=1,2···N Among them, k x,y (u,v) represents the transformation kernel; x, y are the horizontal and vertical coordinates of the point in the point cloud; u, v are the horizontal and vertical coordinates of the pixel in the image; δ represents the Dirac function, which is used as an indicator function in sparse mapping; i represents the point cloud label; j represents the pixel label in the image; N represents the upper limit of the label; δ (x,y)(xi,yi) Used to indicate whether a point (x, y) in the point cloud coincides with a point (xi, yi) in the point cloud; δ (u,v)(uj,vj) Indicates whether the pixel coordinates (u, v) coincide with the pixel coordinates (uj, vj) after point cloud mapping.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the 3D object detection method according to any one of claims 1 to 6 are implemented.

8. A terminal comprising a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, characterized in that: When the processor runs the computer instructions, the steps of the 3D object detection method according to any one of claims 1 to 6 are performed.

Images