Target detection method and device based on color point cloud, electronic equipment and medium

Abstract

The invention relates to a target detection method and device based on color point cloud, electronic equipment and a medium. The method comprises the steps of obtaining color point cloud data based on laser radar data and RGB image data of a target detection area; performing voxelization processing on the color point cloud data to obtain structured data, and performing compression and feature extraction operation on the structured data by using a preset three-dimensional encoder network to obtain a feature tensor of a preset dimension; and generating a top view based on the feature tensor of the preset dimension, and obtaining a target thermodynamic diagram by using a preset central point detection strategy and a preset size regression strategy based on the top view, thereby obtaining central point coordinate information and state information of each target object in the target detection area. Therefore, target detection is carried out by fusing the camera data and the laser radar data, the problem that the target detection precision is low in a complex environment only depending on the laser radar data is solved, and the accuracy and robustness of target detection are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular relates to a target detection method and device based on color point cloud, electronic equipment and medium. BACKGROUND

[0002] The environment perception technology of the unmanned vehicle mainly realizes the detection of the surrounding environment by means of external sensors, such as laser radar, camera, millimeter wave radar, etc., so as to ensure that the unmanned vehicle can timely and accurately perceive the safety hazards existing in the road environment, and quickly take measures to avoid traffic accidents. The environment perception is equivalent to the eyes of the unmanned vehicle, and plays an irreplaceable role in ensuring the safe driving of the unmanned vehicle.

[0003] In the related art, the target detection algorithm based on the laser radar can divide the 3D space into discrete voxel units, and then apply a deep learning model for feature extraction and classification.

[0004] However, this method only relies on laser radar data for target detection, and the target detection accuracy is low in complex environments, which needs to be solved urgently. SUMMARY

[0005] The present application provides a target detection method and device based on color point cloud, electronic equipment and medium, to solve the problem of low target detection accuracy in complex environments only relying on laser radar data, and improve the accuracy and robustness of target detection.

[0006] To achieve the above purpose, the first aspect of the present application provides a target detection method based on color point cloud, comprising the following steps: Obtain laser radar data and RGB image data of a target detection region, and obtain color point cloud data based on the laser radar data and RGB image data; Perform voxelization processing on the color point cloud data to obtain structured data, and use a preset three-dimensional encoder network to perform compression and feature extraction operations on the structured data to obtain a feature tensor of a preset dimension; Generate an overhead view based on the feature tensor of the preset dimension, and use a preset center point detection strategy and a preset size regression strategy to obtain a target heat map based on the overhead view, and obtain the center point coordinate information and state information of each target object in the target detection region based on the target heat map.

[0007] By means of the above technical means, not only the spatial information provided by the laser radar data is utilized, but also the color information provided by the RGB image data is combined, so that the color point cloud data can more comprehensively reflect the characteristics of the target object, the voxelization processing converts the color point cloud data into structured data, which is convenient for subsequent processing, and the accurate detection of the target object in the target detection area is realized.

[0008] According to an embodiment of the present application, the color point cloud data is obtained based on the laser radar data and the RGB image data, comprising: obtaining first internal parameters and first external parameters of the laser radar, and obtaining second internal parameters and second external parameters of the camera; Based on the first internal parameters, the first external parameters, the second internal parameters and the second external parameters, the three-dimensional coordinates corresponding to the laser radar data are projected to the two-dimensional coordinate system corresponding to the RGB image data, and based on a preset pixel-level RGB assignment strategy, each laser radar data projected to the two-dimensional coordinate system corresponding to the RGB image data is assigned with the RGB value of the corresponding pixel, and the color point cloud data is generated.

[0009] By means of the above technical means, the effective fusion of laser radar data and RGB image data is realized, which provides a solid foundation for subsequent feature extraction and target detection.

[0010] According to an embodiment of the present application, the voxelization processing is performed on the color point cloud data to obtain structured data, comprising: Based on a preset spatial resolution condition, the three-dimensional coordinate system of the color point cloud data is discretized into a plurality of cubic units arranged according to a preset arrangement rule, and the structured data is obtained based on the plurality of cubic units arranged according to the preset arrangement rule.

[0011] By means of the above technical means, the dimension of the color point cloud data is effectively reduced, the data redundancy is reduced, and the efficiency of data processing is improved. At the same time, the formation of structured data makes the subsequent feature extraction and target detection process more convenient and accurate, and provides strong technical support for target detection tasks in complex scenes.

[0012] According to an embodiment of the present application, the target heat map is obtained based on the overhead view, using a preset center point detection strategy and a preset size regression strategy, comprising: using the preset center point strategy, identifying at least one preset key point in the overhead view; Based on the at least one preset key point, using the preset size regression strategy, predicting the three-dimensional bounding box information of the target object corresponding to each preset key point; Based on the three-dimensional bounding box information, a preset dynamic Gaussian function radius strategy is used to obtain the target heat map.

[0013] Through the above technical means, the position of the target object in the top view can be more accurately positioned, and the three-dimensional size information of the target object can be predicted. This generation method of the target heat map not only improves the accuracy of target detection, but also enhances the stability and robustness of target detection. In a complex and variable scene, the target detection device can quickly identify the target object, providing reliable data support for subsequent target tracking and behavior analysis.

[0014] According to an embodiment of the present application, when obtaining the center point coordinate information and state information of each target object in the target detection area based on the target heat map, it further comprises: Based on the three-dimensional bounding box information of each target object in the target heat map, a feature extraction operation is performed on the three-dimensional center point of each face of each three-dimensional bounding box to obtain the feature value of each three-dimensional center point; Based on a bilinear interpolation strategy, the feature values of each three-dimensional center point are stacked to obtain a feature vector, and the feature vector is input into a preset neural network to obtain the optimized three-dimensional bounding box information of each target object.

[0015] Through the above technical means, the position and size information of the target object in the three-dimensional space can be further accurately obtained. This optimized three-dimensional bounding box information not only improves the accuracy of target detection, but also provides a more accurate data basis for subsequent three-dimensional reconstruction and behavior analysis.

[0016] According to the target detection method based on color point cloud proposed in the embodiments of the present application, color point cloud data can be obtained based on the lidar data and RGB image data of the target detection area; the color point cloud data is voxelized to obtain structured data, and a preset three-dimensional encoder network is used to compress and extract features of the structured data to obtain a feature tensor of a preset dimension; a top view is generated based on the feature tensor of the preset dimension, and based on the top view, a target heat map is obtained using a preset center point detection strategy and a preset size regression strategy, and then the center point coordinate information and state information of each target object in the target detection area are obtained. Therefore, by fusing camera data and lidar data for target detection, the problem of low target detection accuracy in complex environments relying only on lidar data is solved, and the accuracy and robustness of target detection are improved.

[0017] To achieve the above purpose, the second aspect of the present application proposes a target detection device based on color point cloud, comprising: The first obtaining module is configured to obtain laser radar data and RGB image data of a target detection region, and obtain color point cloud data based on the laser radar data and the RGB image data. The processing module is configured to perform voxelization processing on the color point cloud data to obtain structured data, and perform compression and feature extraction operations on the structured data by using a preset three-dimensional encoder network to obtain a feature tensor of a preset dimension. The second obtaining module is configured to generate a top view based on the feature tensor of the preset dimension, obtain a target heat map by using a preset center point detection strategy and a preset size regression strategy based on the top view, and obtain center point coordinate information and state information of each target object in the target detection region based on the target heat map.

[0018] According to an embodiment of the present application, the first obtaining module is specifically configured to: obtain first internal parameters and first external parameters of a laser radar, and obtain second internal parameters and second external parameters of a camera; project three-dimensional coordinates corresponding to the laser radar data to a two-dimensional coordinate system corresponding to the RGB image data based on the first internal parameters, the first external parameters, the second internal parameters, and the second external parameters, and assign an RGB value of a corresponding pixel to each laser radar data projected to the two-dimensional coordinate system corresponding to the RGB image data based on a preset pixel-level RGB assignment strategy to generate the color point cloud data.

[0019] According to an embodiment of the present application, the processing module is specifically configured to: discretize a three-dimensional coordinate system of the color point cloud data into a plurality of cubic units arranged according to a preset arrangement rule based on a preset spatial resolution condition, and obtain the structured data based on the plurality of cubic units arranged according to the preset arrangement rule.

[0020] According to an embodiment of the present application, the second obtaining module is specifically configured to: identify at least one preset key point in the top view by using the preset center point strategy; predict three-dimensional bounding box information of a target object corresponding to each preset key point by using the preset size regression strategy based on the at least one preset key point; obtain the target heat map by using a preset dynamic Gaussian function radius strategy based on the three-dimensional bounding box information.

[0021] According to an embodiment of the present application, when the center point coordinate information and the state information of each target object in the target detection region are obtained based on the target heat map, the second obtaining module is further configured to: Based on the three-dimensional bounding box information of each target object in the target heat map, a feature extraction operation is performed on a three-dimensional center point of each face of each three-dimensional bounding box to obtain a feature value of each three-dimensional center point; Based on a bilinear interpolation strategy, a stacking operation is performed on the feature value of each three-dimensional center point to obtain a feature vector, and the feature vector is input into a preset neural network to obtain optimized three-dimensional bounding box information of each target object.

[0022] According to the target detection device based on color point cloud provided in the embodiments of the present application, color point cloud data can be obtained based on the laser radar data and the RGB image data of the target detection region; the color point cloud data is voxelized to obtain structured data, and a preset three-dimensional encoder network is used to compress and extract features of the structured data to obtain a feature tensor of a preset dimension; a top view is generated based on the feature tensor of the preset dimension, and based on the top view, a target heat map is obtained by using a preset center point detection strategy and a preset size regression strategy, and then the center point coordinate information and the state information of each target object in the target detection region are obtained. In this way, target detection is performed by fusing camera data and laser radar data, solving the problem of low target detection accuracy in a complex environment by relying only on laser radar data, and improving the accuracy and robustness of target detection.

[0023] To achieve the above object, the third aspect of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target detection method based on color point cloud as described in the above embodiments.

[0024] To achieve the above object, the fourth aspect of the present application provides a computer readable storage medium having a computer program stored thereon, which is executed by a processor to implement the target detection method based on color point cloud as described in the above embodiments.

[0025] The additional aspects and advantages of the present application will be partially given in the following description, partially become obvious from the following description, or be known by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0026] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which: Figure 1 A flowchart of a target detection method based on color point cloud according to an embodiment of the present application is provided. Figure 2 A flowchart of another target detection method based on color point cloud according to an embodiment of the present application is provided. Figure 3 A block schematic diagram of a target detection device based on color point cloud according to an embodiment of the present application is provided. Figure 4 A structural schematic diagram of an electronic device according to an embodiment of the present application is provided.

[0027] Reference signs: 10-target detection device based on color point cloud, 100-first obtaining module, 200-processing module, 300-second obtaining module, 401-memory, 402-processor, 403-communication interface. DETAILED DESCRIPTION

[0028] Embodiments of the present application are described in detail below with reference to the accompanying drawings, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0029] The target detection method based on color point cloud, device, electronic device and medium according to embodiments of the present application are described below with reference to the accompanying drawings. First, the target detection method based on color point cloud according to embodiments of the present application is described with reference to the accompanying drawings.

[0030] Figure 1 is a flowchart of the target detection method based on color point cloud of one embodiment of the present application.

[0031] Exemplarily, as shown in Figure 1 the target detection method based on color point cloud includes the following steps: In step S101, laser radar data and RGB image data of a target detection region are obtained, and color point cloud data is obtained based on the laser radar data and the RGB image data.

[0032] Specifically, in order to realize accurate recognition of the target detection region, in the embodiments of the present application, the laser radar can be used to scan the target detection region to generate three-dimensional point cloud data (i.e. laser radar data), which includes the spatial coordinate information of each point. At the same time, the same target detection region can be photographed by using the camera to generate RGB image data, which includes the color information of each pixel. By combining these two types of data, color point cloud data can be further processed and generated, thereby providing rich information for subsequent target detection and analysis.

[0033] It can be understood that by applying color point cloud for target detection, each point in the color point cloud not only contains coordinate information in three-dimensional space, but also has RGB color information attached. This point cloud data combined with color information makes the target detection process more efficient and accurate. The use of color point cloud target detection algorithm can bring many benefits: color point cloud fuses data from multiple sensors, making the target detection algorithm effectively integrate multi-sensor data, thereby greatly enhancing the overall perception ability of the system. In complex and variable environments such as urban streets, the addition of color information can help improve detection accuracy. It can help filter out background noise, making the target more prominent. In addition, color information also plays an important role in handling occlusion problems. It can help the algorithm better recognize and distinguish occluded targets. For objects with sparse point clouds due to long distance or small volume, color information provides additional visual cues, which are crucial for improving detection accuracy, allowing the algorithm to maintain high detection performance even when facing objects with sparse point clouds.

[0034] For ease of understanding, how to obtain color point cloud data based on lidar data and RGB image data is described in detail below.

[0035] As a possible implementation, in some embodiments, obtaining color point cloud data based on lidar data and RGB image data includes: obtaining first internal parameters and first external parameters of the lidar, and obtaining second internal parameters and second external parameters of the camera; projecting three-dimensional coordinates corresponding to the lidar data to a two-dimensional coordinate system corresponding to the RGB image data based on the first internal parameters, the first external parameters, the second internal parameters, and the second external parameters, and assigning each lidar data projected to the two-dimensional coordinate system corresponding to the RGB image data with an RGB value of a corresponding pixel based on a preset pixel-level RGB assignment strategy to generate color point cloud data.

[0036] Specifically, in the process of obtaining color point cloud data based on lidar data and RGB image data, first, the internal parameters (i.e., first internal parameters) and external parameters (i.e., first external parameters) of the lidar device can be obtained. At the same time, the internal parameters (i.e., second internal parameters) and external parameters (i.e., second external parameters) of the camera device can also be obtained. After mastering these key parameters, the point cloud data captured by the lidar in the three-dimensional space can be mapped into the two-dimensional RGB image data coordinate system through mathematical transformation, and then the RGB value of each point cloud mapped onto the RGB image data corresponding pixel is extracted, and through this process, color point cloud data can be obtained. Color point cloud data not only retains the original three-dimensional space information, but also contains rich color information, thereby providing more intuitive and rich data support for subsequent image processing and analysis. If the point cloud is occluded (such as projected outside the image edge), it can be processed by interpolation or discarded.

[0037] It can be understood that the internal parameters are used to correct the distortion of the sensor itself, the lidar internal parameters include scanning angle resolution, distance accuracy, etc.; the camera internal parameters include focal length, principal point coordinates, distortion coefficients, etc.; the external parameters are used to convert the lidar data and the RGB image data to the same coordinate system (i.e., coordinate system alignment), so that the coordinate systems of the two are aligned. For example, the 3D point cloud coordinate system of the lidar needs to be converted to the 2D image coordinate system of the camera.

[0038] In step S102, the color point cloud data is voxelized to obtain structured data, and a preset three-dimensional encoder network is used to compress and extract features of the structured data to obtain a feature tensor of a preset dimension.

[0039] That is, after obtaining the color point cloud data, the originally irregularly distributed point cloud data can be converted into structured data through voxelization. Then, a preset three-dimensional encoder network (such as a three-dimensional encoder network in related technology) can be used to compress and extract features of the structured data, and finally the high-dimensional point cloud data is converted into a lower-dimensional feature representation (i.e., a feature tensor of a preset dimension) for subsequent processing.

[0040] Next, how to voxelize the color point cloud data to obtain structured data will be described in detail.

[0041] As a possible implementation manner, in some embodiments, voxelizing the color point cloud data to obtain structured data includes: discretizing a three-dimensional coordinate system of the color point cloud data into a plurality of cubic units arranged according to a preset arrangement rule based on a preset spatial resolution condition, and obtaining the structured data based on the plurality of cubic units arranged according to the preset arrangement rule.

[0042] It can be understood that voxelization is a method of dividing a three-dimensional space into small cubes (voxels), similar to pixels in a two-dimensional image. Specifically, when voxelizing color point cloud data, the three-dimensional coordinate system in which the color point cloud data is located can be discretized based on a predetermined spatial resolution condition (which can be calibrated), i.e., the entire three-dimensional space is divided into many small cubic units (i.e., voxels), which are arranged according to certain arrangement rules, and each cubic unit contains a small spatial region. Through such processing, the originally continuous color point cloud data can be converted into a series of discrete cubic units, each of which contains corresponding color and position information. Ultimately, based on these cubic units arranged according to the predetermined arrangement rules, structured data can be obtained, which makes subsequent data processing and analysis more convenient and efficient.

[0043] In step S103, a top view is generated based on the feature tensor of the predetermined dimension, and a target heat map is obtained based on the top view using a preset center point detection strategy and a preset size regression strategy. Based on the target heat map, the center point coordinate information and state information of each target object in the target detection region are obtained.

[0044] Specifically, after obtaining the low-dimensional feature representation (i.e., the feature tensor of the predetermined dimension), a related algorithm or a related neural network model can be used to generate a top view from the feature tensor of the predetermined dimension. After obtaining the top view, the top view can be further processed using a preset center point detection strategy and a preset size regression strategy to obtain a target heat map. The target heat map refers to a two-dimensional image representing the probability of the existence of a target using color or brightness, and each local maximum in the target heat map corresponds to the center point of a target object (i.e., a vehicle).

[0045] The following will explain in detail how to obtain a target heat map based on a top view using a preset center point detection strategy and a preset size regression strategy.

[0046] As a possible implementation, in some embodiments, obtaining a target heat map based on a top view using a preset center point detection strategy and a preset size regression strategy includes: identifying at least one preset key point in the top view using a preset center point strategy; predicting the three-dimensional bounding box information of the target object corresponding to each preset key point based on the at least one preset key point using a preset size regression strategy; and obtaining a target heat map based on the three-dimensional bounding box information using a preset dynamic Gaussian function radius strategy.

[0047] The preset key point is the center point of the target object. The three-dimensional bounding box is a rectangular box in a three-dimensional space, which is used to accurately describe the position and size of the target object.

[0048] Specifically, the use of image-based key point detection technology (i.e., the preset center point detection strategy) can effectively identify the center point (i.e., at least one preset key point) of each object in the overhead view; then, for each identified object center point, a size regression operation can be further performed based on the preset size regression strategy to infer the complete three-dimensional bounding box information of the target object, including the center point coordinate information (i.e., center position) and status information (i.e., length, width, height, heading angle, and speed, etc.) of the target object; finally, to solve the problem that target objects such as vehicles occupy a small area in the map view, based on these three-dimensional bounding box information, the preset dynamic Gaussian function radius strategy can be used to increase the positive supervision of the heat map, thereby generating a target heat map.

[0049] It is understood that the radius of the Gaussian function can be dynamically adjusted based on the size and shape of the target object to better adapt to targets of different sizes. For example, for each predicted target object, based on its 3D bounding box information, a Gaussian distribution hotspot area is generated on the heat map, and its radius is dynamically adjusted based on the size of the target object.

[0050] Furthermore, in some embodiments, when obtaining the center point coordinate information and status information of each target object in the target detection area based on the target heat map, it also includes: based on the three-dimensional bounding box information of each target object in the target heat map, performing a feature extraction operation on the three-dimensional center point of each face of each three-dimensional bounding box to obtain the eigenvalue of each three-dimensional center point; based on the bilinear interpolation strategy, stacking the eigenvalue of each three-dimensional center point to obtain a feature vector, and inputting the feature vector into a preset neural network to obtain the optimized three-dimensional bounding box information of each target object.

[0051] Specifically, by performing feature extraction on the 3D center point of each face of each 3D bounding box, a feature value for each 3D center point is obtained. Next, using a bilinear interpolation strategy, the feature values ​​of each 3D center point are stacked to form a feature vector. This feature vector is then fed into a pre-defined neural network (such as a multilayer perceptron). After processing and optimization by the network, the optimized 3D bounding box information for each target object is obtained.

[0052] To facilitate those skilled in the art to further understand the target detection method based on color point cloud proposed in the embodiment of the present application, the following is combined with Figure 2 Further elaboration.

[0053] like Figure 2 As shown, the target detection method based on color point cloud may include the following steps: Step S201, input the lidar data and RGB image data.

[0054] Step S202, project the lidar data onto the RGB image data by the intrinsic and extrinsic parameters of the camera and lidar, and assign the RGB values of the corresponding RGB image pixels to the lidar data to generate color point cloud data.

[0055] Step S203, voxelize the color point cloud data, that is, divide the three-dimensional space into small voxel units, each of which contains a number of point cloud data points, to obtain structured data. For example, in the autonomous driving scenario, the space around the vehicle can be divided into voxels with a side length of 0.1 meters or 0.2 meters.

[0056] Step S204, use a three-dimensional backbone network to extract local features of the point cloud data in each voxel to obtain the feature representation of each voxel. Then, further process and aggregate the voxel features through a series of three-dimensional convolution layers, gradually expand the receptive field, and extract higher-level semantic features to obtain a three-dimensional feature map containing rich semantic information.

[0057] Step S205, flatten the three-dimensional feature map into an overhead view, use an image-based key point detector to find the center point of the vehicle, and generate a heat map of the vehicle class through a heat map regression method. Each local maximum in the heat map corresponds to a center point of a vehicle; for each detected vehicle center point, the point features from the center position are regressed to obtain other attributes of the vehicle (i.e., three-dimensional bounding box information), including three-dimensional dimensions (length, width, and height), three-dimensional direction (heading angle), and speed, etc. For example, the regression head predicts that the length of the vehicle is 4.5 meters, the width is 1.8 meters, the height is 1.5 meters, the current heading angle is 30 degrees, and the speed is 10 meters / second, etc.

[0058] Step S206, according to the predicted vehicle bounding box information, extract a point feature from the three-dimensional center of each face of the predicted bounding box. For each point, use bilinear interpolation to extract a feature from the feature map output by the backbone network, and input these point features into a fully connected network for refinement to obtain more accurate vehicle position and size information, while predicting a confidence score representing the 3D IoU (Intersection over Union) between the predicted result and the true value. 3DIoU is an index used to evaluate the degree of overlap between the predicted bounding box and the true bounding box in three-dimensional space.

[0059] In step S207, the center point position, three-dimensional size, three-dimensional direction, speed, confidence score and the like of each detected vehicle are output. These information can be used by the autonomous driving system for further decision and control, such as planning the driving path of the vehicle, avoiding collision, etc.

[0060] According to the target detection method based on color point cloud provided in the embodiments of the present application, color point cloud data can be obtained based on the lidar data and RGB image data of the target detection region; the color point cloud data is voxelized to obtain structured data, and a preset three-dimensional encoder network is used to compress and extract features of the structured data to obtain a feature tensor of a preset dimension; a top view is generated based on the feature tensor of the preset dimension, and a target heat map is obtained based on the top view by using a preset center point detection strategy and a preset size regression strategy, and then the center point coordinate information and state information of each target object in the target detection region are obtained. In this way, target detection is performed by fusing camera data and lidar data, solving the problem of low target detection accuracy in complex environments only relying on lidar data, and improving the accuracy and robustness of target detection.

[0061] Next, the target detection device based on color point cloud provided in the embodiments of the present application is described with reference to the accompanying drawings.

[0062] Figure 3 is a block schematic diagram of the target detection device based on color point cloud according to an embodiment of the present application.

[0063] As shown in Figure 3 , the target detection device based on color point cloud 10 comprises a first obtaining module 100, a processing module 200 and a second obtaining module 300.

[0064] The first obtaining module 100 is configured to obtain lidar data and RGB image data of a target detection region, and obtain color point cloud data based on the lidar data and the RGB image data. The processing module 200 is configured to voxelize the color point cloud data to obtain structured data, and use a preset three-dimensional encoder network to compress and extract features of the structured data to obtain a feature tensor of a preset dimension. The second obtaining module 300 is configured to generate a top view based on the feature tensor of the preset dimension, and obtain a target heat map based on the top view by using a preset center point detection strategy and a preset size regression strategy, and obtain the center point coordinate information and state information of each target object in the target detection region based on the target heat map.

[0065] Optionally, in some embodiments, the first obtaining module 100 is specifically configured to: obtain a first internal parameter and a first external parameter of the lidar, and obtain a second internal parameter and a second external parameter of the camera; project the three-dimensional coordinates corresponding to the lidar data to a two-dimensional coordinate system corresponding to the RGB image data based on the first internal parameter, the first external parameter, the second internal parameter, and the second external parameter, and assign the RGB values of the corresponding pixels to each lidar data projected to the two-dimensional coordinate system corresponding to the RGB image data based on a preset pixel-level RGB assignment strategy, to generate the color point cloud data.

[0066] Optionally, in some embodiments, the processing module 200 is specifically configured to: discretize the three-dimensional coordinate system of the color point cloud data into a plurality of cuboid units arranged according to a preset arrangement rule based on a preset spatial resolution condition, and obtain the structured data based on the plurality of cuboid units arranged according to the preset arrangement rule.

[0067] Optionally, in some embodiments, the second obtaining module 300 is specifically configured to: identify at least one preset key point in the top view using a preset center point strategy; predict the three-dimensional bounding box information of the target object corresponding to each preset key point using a preset size regression strategy based on the at least one preset key point; obtain the target heat map using a preset dynamic Gaussian function radius strategy based on the three-dimensional bounding box information.

[0068] Optionally, in some embodiments, when the center point coordinate information and the state information of each target object in the target detection region are obtained based on the target heat map, the second obtaining module 300 is further configured to: perform a feature extraction operation on the three-dimensional center point of each face of each three-dimensional bounding box based on the three-dimensional bounding box information of each target object in the target heat map, to obtain the feature value of each three-dimensional center point; stack the feature values of each three-dimensional center point based on a bilinear interpolation strategy to obtain a feature vector, and input the feature vector into a preset neural network to obtain the optimized three-dimensional bounding box information of each target object.

[0069] It should be noted that the foregoing explanation and description of the embodiment of the color point cloud-based target detection method also apply to the embodiment of the color point cloud-based target detection device, which will not be described here again.

[0070] According to the target detection device based on color point cloud provided by the embodiment of the application, color point cloud data can be obtained based on the laser radar data and the RGB image data of the target detection area; the color point cloud data is voxelized to obtain structured data, and a preset three-dimensional encoder network is used to compress and extract features of the structured data to obtain a feature tensor of a preset dimension; a top view is generated based on the feature tensor of the preset dimension, and based on the top view, a preset center point detection strategy and a preset size regression strategy are used to obtain a target heat map, and then the center point coordinate information and the state information of each target object in the target detection area are obtained. Therefore, target detection is performed by fusing camera data and laser radar data, the problem of low target detection accuracy in a complex environment by relying only on laser radar data is solved, and the accuracy and robustness of target detection are improved.

[0071] Figure 4 The electronic device provided by the embodiment of the application is shown in a structural schematic diagram. The electronic device can include: The memory 401, the processor 402, and the computer program stored in the memory 401 and executable on the processor 402.

[0072] The processor 402 implements the target detection method based on color point cloud provided in the above embodiments when executing the program.

[0073] Further, the electronic device further includes: The communication interface 403 is used for communication between the memory 401 and the processor 402.

[0074] The memory 401 is used to store the computer program executable on the processor 402.

[0075] The memory 401 can include a high-speed RAM (Random Access Memory, random access memory) memory, and can also include a non-volatile memory, such as at least one disk memory.

[0076] If the memory 401, the processor 402, and the communication interface 403 are independently implemented, the communication interface 403, the memory 401, and the processor 402 can be connected to each other through a bus and complete communication between each other. The bus can be an ISA (Industry Standard Architecture, industry standard architecture) bus, a PCI (Peripheral Component Interconnect, peripheral component interconnect) bus, or an EISA (Extended Industry Standard Architecture, extended industry standard architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 4Only one bus or only one type of bus can exist, however.

[0077] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can complete the communication among each other through an internal interface.

[0078] The processor 402 can be a CPU (Central Processing Unit, central processing unit) or an ASIC (Application Specific Integrated Circuit, application specific integrated circuit) or an integrated circuit configured to implement one or more embodiments of the application.

[0079] The embodiment of the application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the color point cloud based target detection method as above.

[0080] In addition, the terms "first", "second", "third", etc. are used only to describe purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise specifically limited.

[0081] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms is not necessarily for the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

[0082] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A target detection method based on color point cloud, characterized in that: The following steps are involved: Acquire laser radar data and RGB image data of the target detection area, and obtain color point cloud data based on the laser radar data and RGB image data; voxelize the color point cloud data to obtain structured data, and compress and extract features from the structured data using a preset three-dimensional encoder network to obtain a feature tensor of a preset dimension; A bird's-eye view is generated based on the feature tensor of the preset dimension, and based on the bird's-eye view, a target heat map is obtained using a preset center point detection strategy and a preset size regression strategy. Based on the target heat map, the center point coordinate information and status information of each target object in the target detection area are obtained.

2. The method according to claim 1, characterized in that The obtaining of color point cloud data based on the laser radar data and the RGB image data includes: Obtaining a first internal parameter and a first external parameter of the laser radar, and obtaining a second internal parameter and a second external parameter of the camera; Based on the first internal parameter, the first external parameter, the second internal parameter and the second external parameter, the three-dimensional coordinates corresponding to the lidar data are projected to the two-dimensional coordinate system corresponding to the RGB image data, and based on a preset pixel-level RGB assignment strategy, the RGB value of the corresponding pixel is assigned to each lidar data projected to the two-dimensional coordinate system corresponding to the RGB image data to generate the color point cloud data.

3. The method according to claim 1, characterized in that The voxelization processing of the color point cloud data to obtain structured data includes: Based on a preset spatial resolution condition, the three-dimensional coordinate system of the color point cloud data is discretized into a plurality of cubic units arranged according to a preset arrangement rule, and the structured data is obtained based on the plurality of cubic units arranged according to the preset arrangement rule.

4. The method according to claim 1, wherein The target heat map is obtained based on the top view using a preset center point detection strategy and a preset size regression strategy, including: Using the preset center point strategy, identifying at least one preset key point in the top view; Based on the at least one preset key point, using the preset size regression strategy, predicting three-dimensional bounding box information of the target object corresponding to each preset key point; Based on the three-dimensional bounding box information, the target heat map is obtained using a preset dynamic Gaussian function radius strategy.

5. The method according to claim 4, characterized in that When obtaining the center point coordinate information and state information of each target object in the target detection area based on the target heat map, the method further includes: Based on the three-dimensional bounding box information of each target object in the target heat map, a feature extraction operation is performed on the three-dimensional center point of each face of each three-dimensional bounding box to obtain a feature value of each three-dimensional center point; Based on the bilinear interpolation strategy, the eigenvalues ​​of each three-dimensional center point are stacked to obtain a feature vector, and the feature vector is input into a preset neural network to obtain the optimized three-dimensional bounding box information of each target object.

6. A target detection device based on color point cloud, characterized in that: include: A first acquisition module is used to acquire laser radar data and RGB image data of the target detection area, and obtain color point cloud data based on the laser radar data and RGB image data; a processing module, configured to voxelize the color point cloud data to obtain structured data, and perform compression and feature extraction operations on the structured data using a preset three-dimensional encoder network to obtain a feature tensor of a preset dimension; The second acquisition module is used to generate a bird's-eye view based on the feature tensor of the preset dimension, and based on the bird's-eye view, obtain a target heat map using a preset center point detection strategy and a preset size regression strategy, and obtain the center point coordinate information and status information of each target object in the target detection area based on the target heat map.

7. The device according to claim 6, characterized in that The first obtaining module is specifically configured to: Obtaining a first internal parameter and a first external parameter of the laser radar, and obtaining a second internal parameter and a second external parameter of the camera; Based on the first internal parameter, the first external parameter, the second internal parameter and the second external parameter, the three-dimensional coordinates corresponding to the lidar data are projected to the two-dimensional coordinate system corresponding to the RGB image data, and based on a preset pixel-level RGB assignment strategy, the RGB value of the corresponding pixel is assigned to each lidar data projected to the two-dimensional coordinate system corresponding to the RGB image data to generate the color point cloud data.

8. The device according to claim 6, characterized in that The processing module is specifically used to: Based on a preset spatial resolution condition, the three-dimensional coordinate system of the color point cloud data is discretized into a plurality of cubic units arranged according to a preset arrangement rule, and the structured data is obtained based on the plurality of cubic units arranged according to the preset arrangement rule.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target detection method based on color point cloud according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the target detection method based on color point cloud as described in any one of claims 1 to 5.

Images