Data fusion method and device, equipment and medium

Abstract

The invention provides a data fusion method, apparatus and device, and a medium. The method comprises the steps of obtaining point cloud data collected by a laser radar sensor and image data collected by a camera; determining a plurality of target point clouds according to the point cloud data, generating three-dimensional bounding boxes corresponding to the plurality of target point clouds, and extracting global image features of the image data according to the image data; wherein the target point cloud is a dynamic point cloud; according to the three-dimensional bounding box and the global image features, determining regions of interest of the plurality of target point clouds, and extracting image features of the regions of interest; determining fusion features of the point cloud data according to the image features of the region of interest and the target point cloud; and determining target fusion data of the point cloud data and the image data according to the fusion features of the point cloud data and a preset aerial view angle plane, so as to improve the detection efficiency and detection precision of the dynamic target based on the target fusion data of the point cloud data and the image data.

Description

Technical Field

[0001] The present application relates to the field of fusion detection technology, and in particular to a data fusion method, device, equipment and medium. Background Art

[0002] In the field of fusion detection, it is usually performed on the vehicle side by fusion target detection of sensors such as images, lidar, and millimeter waves. Combined with the application scenarios of various on-board sensors, an occupancy grid or bird's eye view (BEV) is used to detect targets around the vehicle body.

[0003] At present, when performing real-time vehicle detection through fusion target detection, the point cloud and image fusion detection network usually directly performs network inference on the entire image or point cloud. Due to the differences in sensor structural characteristics, the fusion network often has a large parameter scale, and an overly deep network structure will affect the algorithm's computational efficiency. In addition, the detection distance of roadside fixed equipment is large, generally 150 meters to 200 meters, making it difficult for the BEV algorithm to achieve large-scale and high-precision target detection.

[0004] Therefore, the point cloud image fusion target detection algorithm in the existing technology has certain limitations. Summary of the invention

[0005] The purpose of this application is to provide a data fusion method, device, equipment and medium to address the practical needs of the point cloud image fusion target detection algorithm in the prior art, in view of the deficiencies in the above-mentioned prior art.

[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a data fusion method, the method comprising:

[0008] Obtain point cloud data collected by the lidar sensor and image data collected by the camera;

[0009] According to the point cloud data, a plurality of target point clouds are determined and three-dimensional bounding boxes corresponding to the plurality of target point clouds are generated, and according to the image data, global image features of the image data are extracted; wherein the target point cloud is a dynamic point cloud;

[0010] Determine a region of interest of the plurality of target point clouds according to the three-dimensional bounding box and the global image feature, and extract image features of the region of interest;

[0011] Determining fusion features of the point cloud data according to the image features of the region of interest and the target point cloud;

[0012] According to the fusion characteristics of the point cloud data and a preset bird's-eye view plane, target fusion data of the point cloud data and the image data is determined.

[0013] As an optional implementation, determining a plurality of target point clouds and generating three-dimensional bounding boxes corresponding to the plurality of target point clouds according to the point cloud data includes:

[0014] Performing background filtering processing on the point cloud data to determine whether the voxel where each point cloud in the point cloud data is located is a background voxel;

[0015] If not, clustering the two-dimensional point cloud from a bird's-eye view perspective is performed to take the point cloud as a target point cloud, and to generate a three-dimensional bounding box corresponding to the target point cloud.

[0016] As an optional implementation, before performing background filtering on the point cloud data to determine whether the voxel where each point cloud in the point cloud data is located is a background voxel, the method further includes:

[0017] Performing point cloud detection on the point cloud data to determine a static point cloud in the point cloud data;

[0018] The background voxels are constructed according to the static point clouds.

[0019] As an optional implementation manner, determining the region of interest of the plurality of target point clouds according to the three-dimensional bounding box and the global image feature includes:

[0020] Determine a mask image of a region of interest, wherein the mask image is used to mark the region of interest;

[0021] The three-dimensional bounding box is projected onto the two-dimensional plane of the mask image to determine regions of interest of the multiple target point clouds.

[0022] As an optional implementation manner, extracting the image features of the region of interest includes:

[0023] Enlarging the regions of interest of the multiple target point clouds according to a preset ratio to obtain enlarged regions of interest of the multiple target point clouds;

[0024] The image features of the enlarged region of interest are retained in the global image features, and the image features of the enlarged region of interest are extracted based on a two-dimensional sparse convolution to obtain the image features of the region of interest.

[0025] As an optional implementation, determining the fusion features of the point cloud data according to the image features of the region of interest and the target point cloud includes:

[0026] Based on a preset intrinsic parameter matrix and extrinsic parameter matrix, the image features of the region of interest are back-projected to the region of interest of the target point cloud, and the corresponding relationship between the voxel centers of the multiple target point clouds and the image data is determined;

[0027] The fusion feature of the point cloud data is determined according to the correspondence between the voxel centers of the multiple target point clouds and the image data.

[0028] As an optional implementation, determining target fusion data of the point cloud data and the image data according to the fusion characteristics of the point cloud data and a preset bird's-eye view plane includes:

[0029] The fusion features of the point cloud data are projected onto the preset bird's-eye view plane, and the fusion features on the bird's-eye view plane are processed based on multiple sparse convolutions to obtain the fusion features of a two-dimensional bird's-eye view plane, and the fusion features of the two-dimensional bird's-eye view plane are used as target fusion data of the point cloud data and the image data.

[0030] In a second aspect, an embodiment of the present application provides a data fusion device, the device comprising:

[0031] An acquisition module is used to acquire point cloud data collected by the lidar sensor and image data collected by the camera;

[0032] A determination module, configured to determine a plurality of target point clouds according to the point cloud data and generate three-dimensional bounding boxes corresponding to the plurality of target point clouds, and to extract global image features of the image data according to the image data; wherein the target point cloud is a dynamic point cloud;

[0033] The determination module is further used to determine the region of interest of the plurality of target point clouds according to the three-dimensional bounding box and the global image feature, and extract the image feature of the region of interest;

[0034] The determination module is further used to determine the fusion features of the point cloud data according to the image features of the region of interest and the target point cloud;

[0035] The determination module is further used to determine target fusion data of the point cloud data and the image data according to the fusion characteristics of the point cloud data and a preset bird's-eye view plane.

[0036] As an optional implementation manner, the determining module is specifically used for:

[0037] Performing background filtering processing on the point cloud data to determine whether the voxel where each point cloud in the point cloud data is located is a background voxel;

[0038] If not, clustering the two-dimensional point cloud from a bird's-eye view perspective is performed to take the point cloud as a target point cloud, and to generate a three-dimensional bounding box corresponding to the target point cloud.

[0039] As an optional implementation manner, the determining module is further used to:

[0040] Performing point cloud detection on the point cloud data to determine a static point cloud in the point cloud data;

[0041] The background voxels are constructed according to the static point clouds.

[0042] As an optional implementation manner, the determining module is specifically used for:

[0043] Determine a mask image of a region of interest, wherein the mask image is used to mark the region of interest;

[0044] The three-dimensional bounding box is projected onto the two-dimensional plane of the mask image to determine regions of interest of the multiple target point clouds.

[0045] As an optional implementation manner, the determining module is specifically used for:

[0046] Enlarging the regions of interest of the multiple target point clouds according to a preset ratio to obtain enlarged regions of interest of the multiple target point clouds;

[0047] The image features of the enlarged region of interest are retained in the global image features, and the image features of the enlarged region of interest are extracted based on a two-dimensional sparse convolution to obtain the image features of the region of interest.

[0048] As an optional implementation manner, the determining module is specifically used for:

[0049] Based on a preset intrinsic parameter matrix and extrinsic parameter matrix, the image features of the region of interest are back-projected to the region of interest of the target point cloud, and the corresponding relationship between the voxel centers of the multiple target point clouds and the image data is determined;

[0050] The fusion feature of the point cloud data is determined according to the correspondence between the voxel centers of the multiple target point clouds and the image data.

[0051] As an optional implementation manner, the determining module is specifically used for:

[0052] The fusion features of the point cloud data are projected onto the preset bird's-eye view plane, and the fusion features on the bird's-eye view plane are processed based on multiple sparse convolutions to obtain the fusion features of a two-dimensional bird's-eye view plane, and the fusion features of the two-dimensional bird's-eye view plane are used as target fusion data of the point cloud data and the image data.

[0053] In a third aspect, an embodiment of the present application provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the data fusion method described in the first aspect above.

[0054] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data fusion method described in the first aspect are executed.

[0055] The beneficial effects of this application are:

[0056] The present application provides a data fusion method, device, equipment and medium. In the pre-processing stage of data fusion, multiple dynamic target point clouds are screened from the point cloud data collected by the laser radar sensor, and three-dimensional bounding boxes corresponding to the multiple dynamic target point clouds are generated, and the global image features of the image data are extracted from the image data collected by the camera. Based on the three-dimensional bounding boxes and image data corresponding to each dynamic target point cloud, the region of interest of each dynamic target point cloud is projected to identify the local specific objects and details of the region of interest in the image data, and the image features of the region of interest are extracted based on the global image features and the region of interest of each dynamic target point cloud. According to the image features of the region of interest and the dynamic target point cloud, the fusion features of the point cloud data are fused to obtain the fusion features of the point cloud data, and the fusion features of the point cloud data are three-dimensional fusion features to prepare for the characterization of dynamic moving objects. In the post-processing stage of data fusion, according to the fusion features of the point cloud data and the preset bird's-eye view plane, the fusion features of the point cloud data in the three-dimensional space are converted to the two-dimensional bird's-eye view scene through the perspective conversion, and more complex features are extracted based on the fusion features of the point cloud data to obtain the target fusion data of the point cloud data and the image data. The fusion features of point cloud data are the fusion features of dynamic target point clouds, which makes the network's cropping space larger and improves the fusion efficiency by reducing the network depth or size, thereby improving the detection efficiency and detection accuracy of dynamic target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 A flowchart of a data fusion method provided in an embodiment of the present application;

[0059] Figure 2 A schematic diagram of a process of determining multiple target point clouds and generating three-dimensional bounding boxes corresponding to the multiple target point clouds according to the data fusion method provided in an embodiment of the present application;

[0060] Figure 3 Another schematic diagram of a process of determining multiple target point clouds and generating three-dimensional bounding boxes corresponding to the multiple target point clouds in the data fusion method provided in an embodiment of the present application;

[0061] Figure 4 A schematic diagram of the process of constructing background voxels in the data fusion method provided in an embodiment of the present application;

[0062] Figure 5 Another schematic diagram of the process of constructing background voxels in the data fusion method provided in an embodiment of the present application;

[0063] Figure 6 A schematic diagram of a process for determining regions of interest of multiple target point clouds in a data fusion method provided in an embodiment of the present application;

[0064] Figure 7 A schematic diagram of a process for extracting image features of a region of interest in a data fusion method provided in an embodiment of the present application;

[0065] Figure 8 A schematic diagram of a flow chart of determining fusion features of point cloud data in a data fusion method provided in an embodiment of the present application;

[0066] Fig. 9 A module structure diagram of a data fusion device provided in an embodiment of the present application;

[0067] Fig.10 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0069] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0070] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0071] At present, when detecting objects around the vehicle body in real time, the entire image or point cloud is directly inferred through the point cloud and image fusion detection network. Due to the differences in sensor structural characteristics, the parameter scale of the fusion network is large, and the too deep network structure affects the calculation efficiency of the algorithm. In addition, the detection distance of roadside fixed equipment is large, which makes it difficult for the BEV algorithm to achieve high-precision target detection over a large range, which has certain limitations.

[0072] Based on the above problems, the embodiment of the present application proposes a data fusion method, which utilizes the characteristics of roadside equipment being fixed and seldom moving, and designs a lightweight algorithm to filter out most of the background point clouds, retain the dynamic point clouds, and back-project the image features of the area of ​​interest of the dynamic point clouds back to the area of ​​interest of the dynamic point clouds, thereby obtaining target fusion data of the point cloud data and the image data, so that the real-time detection network pays more attention to the rapid detection of dynamic targets, thereby improving the detection efficiency and detection accuracy.

[0073] Figure 1 The flowchart of the data fusion method provided in the embodiment of the present application is shown in FIG. 1 , and the execution subject of the method can be any computer device with computing processing capabilities. Figure 1As shown, the method includes:

[0074] S101, obtaining point cloud data collected by a laser radar sensor and image data collected by a camera.

[0075] Optionally, during the driving process of the vehicle, the on-board laser radar sensor collects three-dimensional point cloud data, including point clouds of dynamic moving objects such as pedestrians, vehicles and cyclists, and static fixed objects such as roadside equipment; at the same time, the on-board camera collects two-dimensional image data. Among them, the point cloud data and image data are both input data for data fusion. The point cloud data contains three-dimensional geometric information, and the image data contains two-dimensional color, shape and texture information, so as to obtain the target fusion data of the point cloud data and the image data based on the three-dimensional point cloud data and the two-dimensional image data.

[0076] S102, determining multiple target point clouds based on the point cloud data and generating three-dimensional bounding boxes corresponding to the multiple target point clouds, and extracting global image features of the image data based on the image data; wherein the target point cloud is a dynamic point cloud.

[0077] Optionally, the data fusion of point cloud data and image data includes a pre-processing stage and a post-processing stage. In the pre-processing stage of data fusion, it is mainly divided into a point cloud branch and an image branch. In the point cloud branch, multiple dynamic target point clouds are screened from the input point cloud data, such as the point clouds of dynamic moving objects such as dynamic pedestrians, vehicles and cyclists in the aforementioned embodiment, and a three-dimensional bounding box corresponding to each dynamic target point cloud is generated at the same time, wherein the three-dimensional bounding box corresponding to the target point cloud is a rectangular box to surround the dynamic moving object and mark the position and size of the moving object. The three-dimensional bounding box corresponding to the target point cloud can be used for target detection and target tracking of dynamic targets.

[0078] In the image branch, based on the input image data, the global image features of the image data are extracted through image feature extraction processing. Among them, the global image features can represent the features on the entire image relative to the local image features to describe the overall features of the image such as color, shape and texture.

[0079] S103 . Determine regions of interest of multiple target point clouds according to the three-dimensional bounding box and the global image features, and extract image features of the regions of interest.

[0080] Optionally, based on the three-dimensional bounding box corresponding to each dynamic target point cloud and the image data, the region of interest (ROI) of each dynamic target point cloud in the two-dimensional plane of the image is projected. ROI refers to a sub-region in the image, and the data in the ROI is the data in the image data that is designated to be specially paid attention to or specially processed. Based on the global image features and the ROI of each dynamic target point cloud in the two-dimensional plane of the image, the image features of the ROI are extracted. The image features of the ROI are local image features relative to the global image features, so as to identify specific objects and details of the ROI in the image data.

[0081] S104: Determine fusion features of the point cloud data according to the image features of the region of interest and the target point cloud.

[0082] Optionally, according to the image features of the ROI and the dynamic target point cloud, the fusion features of the point cloud data are fused, wherein the fusion features of the point cloud data are three-dimensional fusion features, including the three-dimensional geometric information of the target point cloud in the three-dimensional space and the two-dimensional color, shape and texture information of the image features. Different from the two-dimensional fusion features, the three-dimensional fusion features of the point cloud data are richer, and the rich three-dimensional fusion features can more accurately characterize the dynamic moving objects.

[0083] S105 . Determine target fusion data of the point cloud data and the image data according to the fusion characteristics of the point cloud data and a preset bird's-eye view plane.

[0084] Optionally, in the post-processing stage of data fusion, according to the fusion features of the point cloud data and the preset BEV plane, the fusion features of the point cloud data in the three-dimensional space are converted to the two-dimensional bird's-eye view scene through perspective conversion, and based on the fusion features of the point cloud data, more complex features are extracted through deep learning to obtain the target fusion data of the point cloud data and the image data. Among them, since the fusion features of the point cloud data are the fusion features of the dynamic target point cloud, the cutting space of the network is increased, and the fusion efficiency can be improved by reducing the network depth or size, thereby improving the detection efficiency and detection accuracy of dynamic target detection.

[0085] In this embodiment, in the pre-processing stage of data fusion, multiple dynamic target point clouds are screened from the point cloud data collected by the laser radar sensor, and three-dimensional bounding boxes corresponding to the multiple dynamic target point clouds are generated, and the global image features of the image data are extracted from the image data collected by the camera. Based on the three-dimensional bounding boxes and image data corresponding to each dynamic target point cloud, the region of interest of each dynamic target point cloud is projected to identify the local specific objects and details of the region of interest in the image data, and the image features of the region of interest are extracted based on the global image features and the region of interest of each dynamic target point cloud. According to the image features of the region of interest and the dynamic target point cloud, the fusion features of the point cloud data are fused to obtain the fusion features of the point cloud data, which are three-dimensional fusion features to prepare for characterizing dynamic moving objects. In the post-processing stage of data fusion, according to the fusion features of the point cloud data and the preset bird's-eye view plane, the fusion features of the point cloud data in the three-dimensional space are converted to the two-dimensional bird's-eye view scene through the perspective conversion, and more complex features are extracted based on the fusion features of the point cloud data to obtain the target fusion data of the point cloud data and the image data. The fusion features of point cloud data are the fusion features of dynamic target point clouds, which makes the network's cropping space larger and improves the fusion efficiency by reducing the network depth or size, thereby improving the detection efficiency and detection accuracy of dynamic target detection.

[0086] The following describes in detail the process of determining multiple target point clouds and generating three-dimensional bounding boxes corresponding to the multiple target point clouds based on point cloud data.

[0087] Figure 2 A schematic diagram of a process of determining multiple target point clouds and generating three-dimensional bounding boxes corresponding to the multiple target point clouds according to the data fusion method provided in an embodiment of the present application, such as Figure 2 As shown, the step of determining multiple target point clouds and generating three-dimensional bounding boxes corresponding to the multiple target point clouds according to the point cloud data in the above step S102 includes:

[0088] S201 , performing background filtering processing on the point cloud data to determine whether the voxel where each point cloud in the point cloud data is located is a background voxel.

[0089] Optionally, in the point cloud branch, a three-dimensional density statistical filtering (3D-DSF) algorithm is used to perform background filtering on the input point cloud data to filter out foreground voxels in the point cloud data, where the foreground voxels are the voxels where the dynamic target point cloud is located, and the background voxels are the voxels where the static point cloud is located.

[0090] Specifically, Figure 3Another flow chart of determining multiple target point clouds and generating three-dimensional bounding boxes corresponding to the multiple target point clouds according to the data fusion method provided in the embodiment of the present application, referring to Figure 3 , using the CUDA cores in the Compute Unified Device Architecture (CUDA) to calculate the index of the voxel where each point cloud in the input point cloud data is located, and perform background filtering on the voxel where each point cloud is located, to determine whether the voxel where each point cloud is located is a background voxel, and to filter out the foreground voxel where the target point cloud is located. Figure 3 The small black cubes in the grid can represent background voxels, and the small white cubes can represent foreground voxels.

[0091] S202: If not, cluster the two-dimensional point cloud from a bird's-eye view, take the point cloud as a target point cloud, and generate a three-dimensional bounding box corresponding to the target point cloud.

[0092] Optionally, if it is determined that the voxel where the point cloud is located is not a background voxel, that is, it is determined that the voxel where the point cloud is located is a foreground voxel, then through the two-dimensional point cloud clustering of BEV, the three-dimensional dynamic point cloud in the forward voxel is used as a target point cloud, and the three-dimensional target point cloud is converted into two-dimensional point cloud data to obtain a three-dimensional bounding box corresponding to the target point cloud. Figure 3 ,If the voxel where the point cloud is located is determined as a foreground voxel through background filtering ,processing, the target point cloud is converted into two-dimensional point cloud data through ,two-dimensional grid clustering of BEV, and a three-dimensional bounding box corresponding to the target point ,cloud is generated.

[0093] In this embodiment, the input point cloud data is subjected to background filtering processing by a three-dimensional density statistical filtering algorithm to determine whether the voxels where each point cloud in the point cloud data is located are background voxels, so as to screen out the foreground voxels where the dynamic target point cloud in the point cloud data is located. If it is determined that the voxel where the point cloud is located is a foreground voxel, the three-dimensional dynamic point cloud in the forward voxel is clustered as a target point cloud through the two-dimensional point cloud clustering from a bird's-eye view, and the three-dimensional target point cloud is converted into two-dimensional point cloud data to obtain a three-dimensional bounding box corresponding to the target point cloud. Through the background filtering processing, the background voxels where the static point cloud in the input point cloud data is located are filtered out, and the foreground voxels where the dynamic target point cloud in the point cloud data is located are retained, so as to avoid point cloud clustering of the background voxels where the static point cloud is located, thereby improving the efficiency of point cloud clustering.

[0094] The following describes in detail the process of performing background filtering on the point cloud data to determine whether the voxel where each point cloud in the point cloud data is located is a background voxel.

[0095] Figure 4 A schematic diagram of the process of constructing background voxels of the data fusion method provided in the embodiment of the present application, such as Figure 4As shown, the step before performing background filtering on the point cloud data in step S201 to determine whether the voxel where each point cloud in the point cloud data is located is a background voxel also includes:

[0096] S401, performing point cloud detection on the point cloud data to determine a static point cloud in the point cloud data.

[0097] Optionally, a point cloud detection network or millimeter wave is used to perform point cloud detection on the input point cloud data, and the dynamic target point cloud surrounded by the detection frame or the dynamic target point cloud near the millimeter wave is filtered out to determine the static point cloud in the point cloud data, so as to reduce the statistical points of dynamic targets such as vehicles, pedestrians, and cyclists. Exemplarily, Figure 5 Another flow chart of constructing background voxels of the data fusion method provided in the embodiment of the present application is shown in FIG. Figure 5 , a non-real-time point cloud detection network or millimeter wave is used to perform point cloud detection on the input point cloud data to determine the static point cloud in the point cloud data.

[0098] S402: constructing background voxels according to the static point clouds.

[0099] Optionally, continue to refer to Figure 5 , each CUDA core constructs each background voxel based on each static point cloud. Specifically, the CUDA core voxelizes the static point cloud, calculates the index of the voxel where the static point cloud is located, and increases the voxel value to construct the background voxel in the grid. Since the statistical points of dynamic targets such as vehicles, pedestrians, and cyclists are reduced, each background voxel is constructed only based on each static point cloud, achieving a higher-precision static point cloud background grid construction effect.

[0100] In this embodiment, a point cloud detection network or millimeter wave is used to perform point cloud detection on the input point cloud data, and the dynamic target point cloud is filtered out to determine the static point cloud in the point cloud data. Each static point cloud is voxelized, the index of the voxel where each static point cloud is located is calculated, and the voxel value is increased to construct each background voxel in the grid. By filtering out the dynamic target point cloud in the point cloud data, the number of statistical points of dynamic targets such as vehicles, pedestrians, and cyclists is reduced, and the construction accuracy of the static point cloud background grid is improved.

[0101] The following describes in detail the process of determining regions of interest of multiple target point clouds based on a three-dimensional bounding box and global image features.

[0102] Figure 6 A schematic diagram of a process for determining regions of interest of multiple target point clouds according to a data fusion method provided in an embodiment of the present application, such as Figure 6 As shown, the step of determining the regions of interest of the plurality of target point clouds according to the three-dimensional bounding box and the global image features in the above step S103 includes:

[0103] S601: Determine a mask image of a region of interest, where the mask image is used to mark the region of interest.

[0104] Optionally, a mask image of the ROI is predetermined by manual labeling or computer automatic recognition. The mask image may be a binary image having the same size as the original image and having a non-zero value only in the ROI of the target point cloud, and is used to mark the ROI.

[0105] S602: Project the three-dimensional bounding box onto the two-dimensional plane of the mask image to determine regions of interest of multiple target point clouds.

[0106] Optionally, the three-dimensional bounding box corresponding to each dynamic target point cloud is projected onto the two-dimensional plane of the Mask image, and the Mask image is used to mark the ROI, and the ROI is separated and extracted from the entire original image to distinguish the ROI in the original image and the background area other than the ROI, thereby determining the two-dimensional ROI of each dynamic target point cloud in the image plane.

[0107] In this embodiment, a mask image of the region of interest is determined in advance, and the three-dimensional bounding box corresponding to each dynamic target point cloud is projected onto the two-dimensional plane of the mask image. The mask image is used to separate and extract the region of interest from the entire original image, thereby determining the region of interest of each dynamic target point cloud in the image plane, thereby improving the accuracy of determining the region of interest of multiple target point clouds.

[0108] The following is a detailed description of the process of extracting image features of the region of interest.

[0109] Figure 7 A schematic diagram of a process for extracting image features of a region of interest according to a data fusion method provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the step of extracting the image features of the region of interest in the above step S103 includes:

[0110] S701 , enlarging regions of interest of multiple target point clouds according to a preset ratio to obtain enlarged regions of interest of multiple target point clouds.

[0111] Optionally, since the calibration error may cause the position of the two-dimensional ROI to be inaccurate, the two-dimensional ROI of each dynamic target point cloud may be slightly enlarged according to a preset ratio to obtain the two-dimensional enlarged ROI of each dynamic target point cloud. Exemplarily, the preset ratio may be 1:1.2, that is, the enlarged ROI is 1.2 times the original ROI.

[0112] S702: retaining the image features of the enlarged region of interest in the global image features, and extracting the image features of the enlarged region of interest based on a two-dimensional sparse convolution to obtain the image features of the region of interest.

[0113] Optionally, the image features of the enlarged ROI portion are retained in the global image features of the image data, and the image features of the enlarged ROI portion are further extracted through a two-dimensional sparse convolution to obtain the image features of the ROI of each dynamic target point cloud. Among them, the two-dimensional sparse convolution can reduce the computing power consumption, focusing on the image feature extraction of the enlarged ROI portion. Compared with the image feature extraction of the original ROI portion, the image feature extraction of the enlarged ROI portion can eliminate the deviation of the image features of the ROI.

[0114] In this embodiment, the two-dimensional region of interest of each dynamic target point cloud is slightly enlarged according to a preset ratio to obtain the two-dimensional enlarged region of interest of each dynamic target point cloud, so as to eliminate the position deviation of the two-dimensional region of interest caused by the calibration error. The image features of the enlarged region of interest are retained in the global image features, and the image features of the enlarged region of interest are further extracted through a two-dimensional sparse convolution to obtain the image features of the region of interest of each dynamic target point cloud. The two-dimensional sparse convolution reduces the computing power consumption, focuses on the image feature extraction of the enlarged region of interest, and eliminates the deviation of the image features of the region of interest of the target point cloud.

[0115] The following describes in detail the process of determining the fusion features of the point cloud data based on the image features of the region of interest and the target point cloud.

[0116] Figure 8 A schematic diagram of a flow chart of determining fusion features of point cloud data in a data fusion method provided in an embodiment of the present application, such as Figure 8 As shown, the step of determining the fusion features of the point cloud data according to the image features of the region of interest and the target point cloud in the above step S104 includes:

[0117] S801 . Based on a preset intrinsic parameter matrix and extrinsic parameter matrix, back-project the image features of the region of interest to the region of interest of the target point cloud, and determine the corresponding relationship between the voxel centers of the target point clouds and the image data.

[0118] Optionally, an internal parameter matrix and an external parameter matrix are extracted in advance based on the intrinsic parameters of the camera and the mapping relationship between the point cloud data collected by the laser radar sensor and the image data collected by the camera. Based on the preset internal and external parameter matrices, the image features of the two-dimensional ROI of each dynamic target point cloud are back-projected back to the three-dimensional ROI of the target point cloud to determine the corresponding relationship between the voxel center of the foreground voxel where each dynamic target point cloud is located and the image data.

[0119] S802: Determine fusion features of the point cloud data according to the correspondence between the voxel centers of the multiple target point clouds and the image data.

[0120] Optionally, according to the corresponding relationship between the voxel center of the foreground voxel where each dynamic target point cloud is located and the image data, the fusion features of the point cloud data are fused. Since the image features of the two-dimensional ROI of each dynamic target point cloud are back-projected back to the three-dimensional ROI of the target point cloud, the fusion features of the point cloud data are three-dimensional fusion features. Compared with the two-dimensional fusion features, the three-dimensional fusion features of the point cloud data are richer and can more accurately characterize dynamic moving objects. Back-projecting the image features of the two-dimensional ROI of each dynamic target point cloud back to the three-dimensional ROI of the target point cloud also avoids consuming the computing power of the depth estimation network and improves the fusion efficiency.

[0121] In this embodiment, based on the preset internal and external parameter matrices, the image features of the two-dimensional region of interest of each dynamic target point cloud are back-projected back to the three-dimensional region of interest of the target point cloud, and the corresponding relationship between the voxel center of the foreground voxel where each dynamic target point cloud is located and the image data is determined; and then, according to the corresponding relationship between the voxel center of the foreground voxel where each dynamic target point cloud is located and the image data, the fusion features of the point cloud data are fused. Since the two-dimensional image features of interest of each dynamic target point cloud are back-projected back to the three-dimensional region of interest of the target point cloud, the fusion features of the point cloud data are three-dimensional fusion features with richer features, so as to more accurately characterize dynamic moving objects, and also avoid the computing power consumption of the depth estimation network, and improve the fusion efficiency.

[0122] As an optional implementation, the step of determining target fusion data of the point cloud data and the image data according to the fusion characteristics of the point cloud data and the preset bird's-eye view plane in the above step S105 includes:

[0123] The fusion features of the point cloud data are projected onto a preset bird's-eye view plane, and the fusion features on the bird's-eye view plane are processed based on multiple sparse convolutions to obtain the fusion features of a two-dimensional bird's-eye view plane. The fusion features of the two-dimensional bird's-eye view plane are used as the target fusion data of the point cloud data and the image data.

[0124] Optionally, in the post-processing stage of data fusion, the three-dimensional fusion features of the point cloud data are projected onto a preset BEV plane through perspective conversion to obtain two-dimensional sparse fusion features. The two-dimensional sparse fusion on the BEV plane is processed based on multiple sparse convolutions to obtain more complex and richer two-dimensional BEV plane fusion features, and the two-dimensional BEV plane fusion features are used as the target fusion data of the point cloud data and the image data.

[0125] In this embodiment, in the post-processing stage of data fusion, the three-dimensional fusion features of the point cloud data are projected onto a preset bird's-eye view plane through perspective conversion to obtain two-dimensional sparse fusion features. The two-dimensional sparse fusion on the bird's-eye view plane is processed based on multiple sparse convolutions, and more complex and richer fusion features of the two-dimensional bird's-eye view plane are extracted through deep learning, and the fusion features of the two-dimensional bird's-eye view plane are used as target fusion data of point cloud data and image data. The fusion features of the target fusion data are enriched, so that high-precision dynamic target detection can be performed based on the target fusion data of point cloud data and image data.

[0126] Based on the same inventive concept, a data fusion device corresponding to the data fusion method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned data fusion method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0127] Fig. 9 The module structure diagram of the data fusion device provided in the embodiment of the present application is as follows: Fig. 9 As shown, the device comprises:

[0128] The acquisition module 901 is used to acquire point cloud data collected by the laser radar sensor and image data collected by the camera.

[0129] The determination module 902 is used to determine multiple target point clouds and generate three-dimensional bounding boxes corresponding to the multiple target point clouds based on the point cloud data, and extract global image features of the image data based on the image data.

[0130] Among them, the target point cloud is a dynamic point cloud;

[0131] The determination module 902 is further used to determine the regions of interest of the multiple target point clouds according to the three-dimensional bounding box and the global image features, and to extract the image features of the regions of interest.

[0132] The determination module 902 is further used to determine the fusion features of the point cloud data according to the image features of the region of interest and the target point cloud.

[0133] The determination module 902 is further used to determine target fusion data of the point cloud data and the image data according to the fusion characteristics of the point cloud data and a preset bird's-eye view plane.

[0134] As an optional implementation, the determination module 902 is specifically configured to:

[0135] Background filtering is performed on the point cloud data to determine whether the voxel where each point cloud in the point cloud data is located is a background voxel.

[0136] If not, the point cloud is clustered from a bird's-eye view and is taken as a target point cloud, and a three-dimensional bounding box corresponding to the target point cloud is generated.

[0137] As an optional implementation manner, the determining module 902 is further configured to:

[0138] Perform point cloud detection on the point cloud data to determine the static point cloud in the point cloud data.

[0139] Based on each static point cloud, each background voxel is constructed.

[0140] As an optional implementation, the determination module 902 is specifically configured to:

[0141] A mask image of a region of interest is determined, where the mask image is used to mark the region of interest.

[0142] The 3D bounding box is projected onto the 2D plane of the mask image to determine the regions of interest of multiple target point clouds.

[0143] As an optional implementation, the determination module 902 is specifically configured to:

[0144] The regions of interest of the multiple target point clouds are enlarged according to a preset ratio to obtain enlarged regions of interest of the multiple target point clouds.

[0145] The image features of the enlarged region of interest are retained in the global image features, and the image features of the enlarged region of interest are extracted based on a two-dimensional sparse convolution to obtain the image features of the region of interest.

[0146] As an optional implementation, the determination module 902 is specifically configured to:

[0147] Based on the preset intrinsic parameter matrix and extrinsic parameter matrix, the image features of the region of interest are back-projected to the region of interest of the target point cloud, and the corresponding relationship between the voxel centers of multiple target point clouds and the image data is determined.

[0148] According to the corresponding relationship between the voxel centers of multiple target point clouds and the image data, the fusion characteristics of the point cloud data are determined.

[0149] As an optional implementation, the determination module 902 is specifically configured to:

[0150] The fusion features of the point cloud data are projected onto a preset bird's-eye view plane, and the fusion features on the bird's-eye view plane are processed based on multiple sparse convolutions to obtain the fusion features of a two-dimensional bird's-eye view plane. The fusion features of the two-dimensional bird's-eye view plane are used as the target fusion data of the point cloud data and the image data.

[0151] The present application also provides a computer device, such as Fig.101 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application, including: a processor 101, a memory 102 and a bus 103. The memory 102 stores machine-readable instructions executable by the processor 101 (for example, Fig. 9 In the device, module 901 is obtained and execution instructions corresponding to module 902 are determined, etc.), when the computer device is running, the processor 101 communicates with the memory 102 through the bus 103, and when the machine-readable instructions are executed by the processor 101, the steps of the data fusion method in the above embodiment are executed.

[0152] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data fusion method in the above embodiment are executed.

[0153] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0154] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0155] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A data fusion method, characterized in that: include: Obtain point cloud data collected by the lidar sensor and image data collected by the camera; According to the point cloud data, a plurality of target point clouds are determined and three-dimensional bounding boxes corresponding to the plurality of target point clouds are generated, and according to the image data, global image features of the image data are extracted; wherein the target point cloud is a dynamic point cloud; Determine a region of interest of the plurality of target point clouds according to the three-dimensional bounding box and the global image feature, and extract image features of the region of interest; Determining fusion features of the point cloud data according to the image features of the region of interest and the target point cloud; According to the fusion characteristics of the point cloud data and a preset bird's-eye view plane, target fusion data of the point cloud data and the image data is determined.

2. The method according to claim 1, characterized in that The step of determining a plurality of target point clouds and generating three-dimensional bounding boxes corresponding to the plurality of target point clouds according to the point cloud data includes: Performing background filtering processing on the point cloud data to determine whether the voxel where each point cloud in the point cloud data is located is a background voxel; If not, clustering the two-dimensional point cloud from a bird's-eye view perspective is performed to take the point cloud as a target point cloud, and to generate a three-dimensional bounding box corresponding to the target point cloud.

3. The method according to claim 2, characterized in that Before performing background filtering on the point cloud data to determine whether the voxel where each point cloud in the point cloud data is located is a background voxel, the method further includes: Performing point cloud detection on the point cloud data to determine a static point cloud in the point cloud data; The background voxels are constructed according to the static point clouds.

4. The method according to claim 1, characterized in that: The determining, according to the three-dimensional bounding box and the global image feature, the region of interest of the plurality of target point clouds comprises: Determine a mask image of a region of interest, wherein the mask image is used to mark the region of interest; The three-dimensional bounding box is projected onto the two-dimensional plane of the mask image to determine regions of interest of the multiple target point clouds.

5. The method according to claim 1, characterized in that The extracting the image features of the region of interest comprises: Enlarging the regions of interest of the multiple target point clouds according to a preset ratio to obtain enlarged regions of interest of the multiple target point clouds; The image features of the enlarged region of interest are retained in the global image features, and the image features of the enlarged region of interest are extracted based on a two-dimensional sparse convolution to obtain the image features of the region of interest.

6. The method according to claim 1, characterized in that The step of determining the fusion features of the point cloud data according to the image features of the region of interest and the target point cloud includes: Based on a preset intrinsic parameter matrix and extrinsic parameter matrix, the image features of the region of interest are back-projected to the region of interest of the target point cloud, and the corresponding relationship between the voxel centers of the multiple target point clouds and the image data is determined; The fusion feature of the point cloud data is determined according to the correspondence between the voxel centers of the multiple target point clouds and the image data.

7. The method according to claim 1, characterized in that The step of determining target fusion data of the point cloud data and the image data according to the fusion characteristics of the point cloud data and a preset bird's-eye view plane includes: The fusion features of the point cloud data are projected onto the preset bird's-eye view plane, and the fusion features on the bird's-eye view plane are processed based on multiple sparse convolutions to obtain the fusion features of a two-dimensional bird's-eye view plane, and the fusion features of the two-dimensional bird's-eye view plane are used as target fusion data of the point cloud data and the image data.

8. A data fusion device, characterized in that: The device comprises: An acquisition module is used to acquire point cloud data collected by the lidar sensor and image data collected by the camera; A determination module, configured to determine a plurality of target point clouds according to the point cloud data and generate three-dimensional bounding boxes corresponding to the plurality of target point clouds, and to extract global image features of the image data according to the image data; wherein the target point cloud is a dynamic point cloud; The determination module is further used to determine the region of interest of the plurality of target point clouds according to the three-dimensional bounding box and the global image feature, and extract the image feature of the region of interest; The determination module is further used to determine the fusion features of the point cloud data according to the image features of the region of interest and the target point cloud; The determination module is further used to determine target fusion data of the point cloud data and the image data according to the fusion characteristics of the point cloud data and a preset bird's-eye view plane.

9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the data fusion method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data fusion method according to any one of claims 1 to 7 are executed.

Images