A 3D target detection method, computer program product and terminal

Abstract

The application discloses a 3D target detection method, a computer program product and a terminal, and belongs to the technical field of image processing. A laser radar and an image acquisition device are jointly calibrated. A point cloud global feature is mapped to an image feature map using a transformation kernel, a sparse matrix is constructed, and the image feature map is flattened into a matrix. After batch normalization processing of the sparse matrix and the matrix, connection processing is performed to obtain fusion features of the image and the point cloud. The fusion features are used for region proposal to generate a candidate target region. The candidate region is classified and a bounding box is regressed to determine the category and position of the target, and a 3D target detection result is obtained. The mapping relationship between the point cloud and the monocular image is constructed by using the sparse point cloud, so that only image pixel point coordinates and point cloud coordinate pairs that share the same points are paired, accurate alignment of the features is realized, the accuracy of target detection is improved, and the precision of 3D target detection is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a 3D target detection method fusing point cloud and image, a computer program product and a terminal. BACKGROUND

[0002] 3D target detection is a technology that uses sensor data (such as images captured by cameras and point cloud data captured by lidars) to identify and locate targets in three-dimensional space. It predicts the position, size and direction of the target, providing key information for autonomous driving, robot navigation, etc. In order to obtain more comprehensive and accurate environmental information, it is often necessary to fuse data from different sensors. Making full use of the advantages of various modalities, designing safe and efficient environmental perception technology becomes a crucial task. In particular, the exploration of multi-modal fusion detection technology will be a key direction in the development of future autonomous driving systems. Multi-modal perception not only improves the perception accuracy, but also provides more accurate and rich information for subsequent path planning and decision-making. For example, the distance data provided by the lidar combined with the object recognition results of the camera can help the system better predict the intentions of other vehicles or pedestrians, thereby optimizing driving decisions and vehicle control. However, image and point cloud data come from different sensors, and they have different characteristics and limitations when capturing environmental information. Images provide rich color and texture information, but lack depth information; while lidar provides accurate three-dimensional spatial information, but is limited in resolution and robustness to environmental light changes. It can be seen that point cloud data and image have different representations and come from different coordinate systems, which leads to the problem of misalignment between radar point cloud and image features in the fusion process, and further leads to inaccurate target positioning, thereby affecting the accuracy of 3D target detection. SUMMARY

[0003] The purpose of the present application is to overcome the problems of the prior art and provide a 3D target detection method, a computer program product and a terminal.

[0004] The purpose of the present application is achieved by the following technical solution: a 3D target detection method, comprising the following steps:

[0005] Jointly calibrate the radar and the image acquisition device;

[0006] Feature extraction and feature aggregation processing are performed on the point cloud data to obtain point cloud global features;

[0007] Feature extraction and sparse non-uniform pooling processing are performed on the image to obtain image feature maps;

[0008] The point cloud global feature is mapped to the image feature map using a transformation kernel, a sparse matrix is constructed, and the image feature map is flattened into a matrix; the sparse matrix and the matrix are connected after batch normalization processing to obtain the fusion feature of the image and the point cloud;

[0009] The fusion feature is used for region proposal to generate a candidate target region;

[0010] The candidate region is classified and the bounding box is regressed to determine the category and position of the target, and a 3D target detection result is obtained.

[0011] In an example, a point cloud segmentation neural network PointNet is used for feature extraction and feature aggregation processing of point cloud data.

[0012] In an example, the feature extraction and feature aggregation processing of the point cloud data includes:

[0013] The point cloud data is processed using PointNet to generate a feature tensor;

[0014] A maximum pooling operation is performed on each feature dimension of the feature tensor on all points to obtain a global feature vector.

[0015] In an example, before the feature extraction and feature aggregation processing of the point cloud data, the method further includes:

[0016] The point cloud data is discretized into a plurality of fixed-size columns on a two-dimensional plane;

[0017] The distance of the arithmetic mean of all points in each column and the offset of each point to the center of the column are calculated, and each point is represented as a nine-dimensional feature vector.

[0018] In an example, after the feature extraction of the image and before the sparse non-uniform pooling processing, the method further includes:

[0019] Target features at different scales are extracted through multiple levels of the convolutional network to realize multi-scale feature fusion processing.

[0020] In an example, the calculation expression of the transformation kernel sparsification is:

[0021] k x,y (u,v)=δ (x,y)(xi,yi) k x,y (u,v)δ (u,v)(uj,vj) ,i,j=1,2…N

[0022] Where k x,y(u,v) represents a transform kernel; x, y are respectively the horizontal and vertical coordinates of the point cloud; u, v are respectively the horizontal and vertical coordinates of the pixel in the image; delta represents the Dirac function, which is used as an indicator function in sparse mapping; i represents the point cloud label; j represents the pixel label in the image; N represents the upper limit value of the label; delta (x,y)(xi,yi) for indicating whether the point (x, t) in the point cloud coincides with the point (xi, yi) in the point cloud; delta (u,v)(uj,vj) for indicating whether the pixel point coordinate (u, v) coincides with the pixel point coordinate (uj, vj) after the point cloud mapping.

[0023] It should be further pointed out that the technical features corresponding to the above examples can be combined or replaced to form new technical solutions.

[0024] The application also includes a computer program product comprising a computer program which, when executed by a processor, implements the steps of the 3D target detection method formed by any one of the above examples or a combination of multiple examples.

[0025] The application also includes a storage medium having computer instructions stored thereon, which, when executed, perform the steps of the 3D target detection method formed by any one of the above examples or a combination of multiple examples.

[0026] The application also includes a terminal comprising a memory and a processor, the memory having computer instructions stored thereon executable on the processor, and the processor executing the computer instructions performing the steps of the 3D target detection method formed by any one of the above examples or a combination of multiple examples.

[0027] Compared with the prior art, the application has the following advantages:

[0028] 1. In an example, joint calibration of radar and image acquisition device can fix the pose relationship of the two, and then the mapping relationship between the point cloud and the monocular image is constructed by using sparse point cloud, which ensures that only image pixel point coordinates and point cloud coordinates that share the same point are paired, thereby realizing accurate alignment of features and improving the accuracy of target detection, thereby ensuring the accuracy of 3D target detection; the sparse non-uniform pooling layer realizes effective conversion and fusion of point cloud and image features through sparse matrix multiplication, and only when corresponding point cloud data exists, pixels in the image can be used for fusion, avoiding unnecessary calculation and potential information loss, and further ensuring the accuracy of target positioning.

[0029] At the same time, the point cloud data provides accurate 3D geometric information of the object, and the monocular camera provides rich visual information such as texture and color. By fusing these two kinds of data, the geometric information and visual information can be combined, so that the accuracy of 3D target detection is higher, especially in complex scenes such as occlusion and low brightness.

[0030] 2. In an example, by performing columnar partitioning and feature enhancement on point cloud data, high-dimensional point cloud data can be effectively processed and fused, better recognizing objects in 3D space, thereby improving the accuracy of detection.

[0031] 3. In an example, by extracting target features at different scales, the perception of target shape and size can be enhanced, further improving the accuracy of 3D target detection. BRIEF DESCRIPTION OF DRAWINGS

[0032] The specific embodiments of the present application will be further described in conjunction with the accompanying drawings, which are used to provide further understanding of the present application, and form a part of the present application. In these drawings, the same reference numerals represent the same or similar parts. The schematic embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application.

[0033] Fig. 1 A method flowchart is provided for an example of the present application.

[0034] Fig. 2 A method framework diagram is provided for an example of the present application. DETAILED DESCRIPTION

[0035] The technical solutions of the present application will be described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0036] In the description of the present application, it should be noted that the directions or positional relationships indicated by "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like are based on the directions or positional relationships described in the drawings, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the ordinal numbers (such as "first and second", "first to fourth", etc.) are used to distinguish objects, and are not limited to the order, and cannot be understood as indicating or implying relative importance.

[0037] In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, the terms of "mounting", "connecting", "connecting" should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected, it can be mechanically connected, or it can be electrically connected, it can be directly connected, or indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0038] In addition, the technical features involved in the different embodiments of the application described below can be combined with each other as long as there is no conflict between them.

[0039] Single sensor is easy to be limited by its own physical characteristics, such as sparse data acquisition of point cloud for long-distance objects, monocular camera lacks depth information. Fusing point cloud and monocular camera data can complement each other's advantages, improve the adaptability to different environments and scenes, and make the detection more robust and reliable. Therefore, the present application proposes a 3D target detection method fusing point cloud and image.

[0040] In an example, as shown in Figs. 1-2 A 3D target detection method fusing point cloud and image, the method comprises the following steps:

[0041] S10: jointly calibrate the radar and the image acquisition device.

[0042] The image acquisition device is a camera, a camera, etc. For camera calibration, first download the template file for camera calibration: calibration board PDF file, print and use the checkerboard calibration method to obtain the intrinsic parameters of the camera. Then, the laser radar and the camera are jointly calibrated to fix the pose relationship between the camera and the radar. First, the camera uses the checkerboard calibration method to obtain the camera intrinsic parameter matrix, and then the camera and the laser radar are jointly calibrated to obtain the extrinsic parameters, i.e. the rotation matrix and the translation vector of the laser radar relative to the camera. The intrinsic parameters are mainly used for image de-warping and pixel-to-camera coordinate conversion, and the extrinsic parameters are used for spatial alignment between different sensor coordinate systems, to ensure accurate projection and fusion of image and point cloud data in BEV.

[0043] S20: feature extraction and feature aggregation processing are performed on the point cloud data to obtain point cloud global features.

[0044] The point cloud segmentation network first extracts shallow features from the original point cloud data, and then extracts feature information fused with global and local features to obtain deep features of the point cloud.

[0045] S30: feature extraction and sparse non-uniform pooling processing are performed on the image to obtain image feature maps.

[0046] The visual feature information of the monocular image is extracted by using a feature extraction network, and then sparse non-uniform pooling processing is performed through a sparse non-uniform pooling layer. The sparse non-uniform pooling layer realizes effective conversion and fusion of features through sparse matrix multiplication. The sparse non-uniform pooling layer flattens the front view (monocular view) feature map into a dense matrix, and obtains the pooled feature map by multiplying it with a sparse matrix. This means that only when the corresponding point cloud data exists, the pixel in the image will be used for fusion, avoiding unnecessary calculation and potential information loss. Due to the sparsity of the point cloud, this calculation is efficient and can maintain the correspondence of the features, ensuring the accuracy of 3D target detection.

[0047] S40: Map the point cloud global feature to the image feature map using the transformation kernel, construct a sparse matrix, and flatten the image feature map into a matrix; after batch normalization processing of the sparse matrix and the matrix, connect them to obtain the fusion feature of the image and the point cloud.

[0048] The mapping relationship between the point cloud and the monocular image is constructed using the sparse point cloud, ensuring that only image pixel point coordinates and point cloud coordinate pairs that share the same point are paired, thereby realizing accurate alignment of features and improving the accuracy of target detection, thereby ensuring the accuracy of 3D target detection.

[0049] Meanwhile, the present application performs data fusion of the point cloud and the monocular image before the region proposal stage, which means that the entire feature map is used for fusion processing, not just the region of interest, thereby providing more rich context information. This method can more effectively utilize multi-view features and reduce the problem of feature misalignment, improving the performance and robustness of 3D target detection.

[0050] S50: Use the fusion feature to perform region proposal to generate candidate target regions. Specifically, a region proposal network (such as RPN) is used to generate candidate regions that may contain targets.

[0051] S60: Classify and bounding box regression on the candidate regions based on the detection head to determine the class and location of the target, and obtain the 3D target detection result.

[0052] Preferably, after the bounding box regression processing, overlapping or redundant detection results are removed through techniques such as non-maximum suppression (NMS), thereby obtaining the final 3D target detection result.

[0053] In an example, a point cloud segmentation neural network PointNet is used for feature extraction and feature aggregation processing of the point cloud data. Specifically, the point cloud segmentation neural network PointNet is used to process the points in each column to generate a feature tensor with a shape of [N, D], where N is the number of points in the point cloud, and D is the feature dimension extracted for each point; a max operation is performed on the feature channels, and a maximum pooling operation is performed on each feature dimension of the feature tensor over all points to obtain a global feature vector with a length of D.

[0054] In an example, before the feature extraction and feature aggregation processing of the point cloud data, the following steps are further included:

[0055] Pillar division: the point cloud data is discretized into a plurality of fixed-size pillars on a two-dimensional plane (x-y plane), and each pillar is a specific two-dimensional grid unit, which is set according to the beam of the laser radar and the number of point clouds in the pillar;

[0056] Feature enhancement: the distance c of the arithmetic mean of all points in each pillar and the offset p of each point to the center of the pillar are calculated; for each point in each pillar, the feature information of the point is enhanced, and the x c ,y c ,z c ,x p and y p of each point in each pillar are increased, and each point is represented as a 9-dimensional feature vector.

[0057] In an example, a monocular image is subjected to feature extraction by a feature extraction network such as a convolutional neural network to obtain rich visual features. Preferably, after the feature extraction of the image, before the sparse non-uniform pooling processing, the following steps are further included:

[0058] Different scale target features are extracted by multiple levels of the convolutional network to realize multi-scale feature fusion processing and ensure that various size target objects in the image are detected.

[0059] In an example, the input size of the monocular image view is H f W f , the size of the bird's eye view (the fusion image of the monocular image and the point cloud data) is L b ×W b , and the size of the radar point cloud is {(x i , y i , z i )|i=1,2,…N}. Then, the calculation expression of the kernel sparsification is:

[0060] k x,y (u,v)=δ (x,y)(xi,yi) k x,y (u,v)δ(u,v)(uj,vj) i,j = 1,2…N

[0061] where k x,y (u,v) represents the transform kernel; x, y are the horizontal and vertical coordinates of the point cloud respectively; u, v are the horizontal and vertical coordinates of the pixel in the image respectively; δ represents the Dirac function, which is used as an indicator function in sparse mapping; i represents the point cloud label; j represents the pixel label in the image; N represents the upper limit value of the label; δ (x,y)(xi,yi) is used to represent whether the point (x, y) in the point cloud coincides with the point (xi, yi) in the point cloud; δ (u,v)(uj,vj) represents whether the pixel point coordinate (u, v) coincides with the pixel point coordinate (uj, vj) after the point cloud mapping. At this time, the feature mapping (such as an image) is taken as the input, and (u i , v i ), (x i , y i ) are taken as parameters to construct an L b W b ×H f W f sparse matrix M; the H f ×W f ×C feature map is flattened into a dense H f W f ×C matrix B = MF; after batch normalization processing of the sparse matrix M and the matrix B, connection processing is performed to obtain the fusion feature of the image and the point cloud.

[0062] In this example, the fusion processing of the image and the point cloud is realized, the 2D image captured by the monocular camera provides detailed information of the object for 3D detection, which helps more accurate object recognition and classification. The point cloud data can accurately estimate the shape and size of the object, combined with the data of the camera, the classification accuracy of target detection can be improved. Further, the point cloud data becomes sparse in long-distance target detection, while the monocular camera can maintain good long-distance resolution. By fusing the data of the two, the shortcomings of the point cloud data in long-distance target detection can be made up, and the detection effect in the long-distance scene can be ensured. In addition, compared with using high-precision laser radar or binocular camera system, the calculation and deployment cost of the monocular camera is lower, and the 2D image data can be processed quickly. Through reasonable multi-modal fusion technology, unnecessary redundant calculation can be reduced, the calculation efficiency can be improved, and the real-time requirement of 3D target detection can be met.

[0063] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the 3D target detection method according to any one of the examples or a combination of multiple examples. The processor can be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the application.

[0064] The application further provides a storage medium having the same inventive concept as the 3D target detection method according to any one of the examples or a combination of multiple examples, and storing computer instructions which, when executed, perform the steps of the 3D target detection method according to any one of the examples or a combination of multiple examples.

[0065] Based on such understanding, the technical solution of the embodiments, essentially or the part that contributes to the prior art, or the part of the technical solution can be embodied in the form of a software product stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0066] The application further provides a terminal having the same inventive concept as any one of the examples or a combination of multiple examples of the 3D target detection method, comprising a memory and a processor, the memory storing computer instructions executable on the processor, and the processor executing the computer instructions to perform the steps of the 3D target detection method. The processor can be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the application.

[0067] In an example, the terminal, i.e., the electronic device, is in the form of a general-purpose computing device, and the components of the electronic device can include but are not limited to the at least one processing unit (processor), the at least one storage unit, and a bus connecting different system components including the storage unit and the processing unit.

[0068] The storage unit stores program codes which can be executed by the processing unit to cause the processing unit to perform the steps according to various exemplary embodiments of the application described in the "Exemplary Method" section of the specification. For example, the processing unit can execute the 3D target detection method.

[0069] The storage can include a readable medium in the form of volatile storage such as a random access memory (RAM) 3201 and / or cache memory and can further include a read only memory (ROM).

[0070] The storage can also include a program / utility, having a set (at least one) of program modules that are configured to carry out the functions of the examples described herein, including an operating system, one or more application programs, other program modules, and program data, each or some combination thereof, which may

[0071] The bus can represent one or more of several types of bus structures, including an external bus or bus controller, a peripheral bus, a graphics acceleration port, a local bus, or any of a variety of busses using any of a variety of bus architectures.

[0072] The electronic device can also communicate with one or more external devices such as a keyboard or a pointing device, through an I / O interface. The electronic device can also communicate to one or more devices that enable a user to interact with it, and / or to one or more devices (e.g., a router, a modem, a server, etc.) that enable it to communicate with one or more other computing devices. Such communication can occur via an I / O interface. Still yet, the electronic device can communicate to one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or the Internet) through a network adapter. It should be appreciated that the electronic device can be a part of a larger system, including but not limited to a network of electronic devices, a cloud computing system, or the like.

[0073] With the above implementations described in detail, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the example embodiments can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the methods of the example embodiments.

[0074] The above detailed description of the application is not intended to limit the application to the specific embodiments described, but is intended to cover all alternatives that fall within the scope of the application.

Claims

1. A 3D object detection method, characterized in that, The method comprises the following steps: Jointly calibrate the radar and the image acquisition device; Perform feature extraction and feature aggregation processing on the point cloud data to obtain global features of the point cloud; Perform feature extraction and sparse non-uniform pooling processing on the image to obtain an image feature map; Map the global features of the point cloud onto the image feature map using a transformation kernel, construct a sparse matrix, and flatten the image feature map into a matrix; Connect the sparse matrix and the matrix after batch normalization processing to obtain fusion features of the image and the point cloud; Use the fusion features to perform region proposal to generate a candidate target region; Classify and regress the bounding box of the candidate region to determine the category and position of the target, and obtain a 3D target detection result. The calculation expression of the transformation kernel sparsification is: k x,y (u,v) = δ (x,y)(xi,yi) k x,y (u,v) δ (u,v)(uj,vj) i,j = 1,2...N wherein k x,y (u, v) represents a transform kernel; x, y are respectively the horizontal and vertical coordinates of the point in the point cloud; u, v are respectively the horizontal and vertical coordinates of the pixel in the image; δ represents a Dirac function, which is used as an indicator function in sparse mapping; i represents the point cloud label; j represents the pixel label in the image; N represents the upper limit value of the label; δ (x,y)(xi,yi) is used to represent whether the point (x, y) in the point cloud coincides with the point (xi, yi) in the point cloud; δ (u,v)(uj,vj) represents whether the pixel point coordinate (u, v) coincides with the pixel point coordinate (uj, vj) after the point cloud mapping.

2. The 3D object detection method of claim 1, wherein, The point cloud data is processed by a point cloud segmentation neural network PointNet to perform feature extraction and feature aggregation processing.

3. The 3D object detection method of claim 2, wherein, The feature extraction and feature aggregation processing on the point cloud data comprises: The point cloud data is processed by PointNet to generate a feature tensor; A maximum pooling operation is performed on each feature dimension of the feature tensor on all points to obtain a global feature vector.

4. The 3D object detection method of claim 1, wherein, Before the feature extraction and feature aggregation processing on the point cloud data, the method further comprises: Discretize the point cloud data into a plurality of fixed-size columns on a two-dimensional plane; Calculate the distance of the arithmetic mean of all points in each column and the offset of each point to the column center, and represent each point as a nine-dimensional feature vector.

5. The 3D object detection method of claim 1, wherein, After the feature extraction on the image and before the sparse non-uniform pooling processing, the method further comprises: Extract target features at different scales through multiple levels of convolutional networks to realize multi-scale feature fusion processing.

6. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the 3D target detection method of any one of claims 1-5.

7. A terminal comprising a memory and a processor, said memory having stored thereon computer instructions executable on said processor, characterized in that, The processor executes the computer instructions to perform the steps of the 3D target detection method of any one of claims 1-5.

Images