Fusion method, 3D target detection method, vehicle-mounted device and storage medium

Abstract

The application discloses a fusion method, a 3D target detection method, a vehicle-mounted device and a storage medium. The fusion method comprises the following steps: acquiring a point cloud and an image used for describing a predetermined scene; performing feature extraction processing on the point cloud to obtain a point cloud bird's-eye view feature map; performing feature extraction processing on the image by using a trained preset deep learning model to obtain an image bird's-eye view feature map; performing alignment processing on the point cloud bird's-eye view feature map and the image bird's-eye view feature map to obtain a point cloud target feature map; and inputting the point cloud target feature map and the image bird's-eye view feature map into a trained fusion deep learning model to obtain a fusion feature map used for describing the predetermined scene. By performing alignment processing on the point cloud bird's-eye view feature map and the image bird's-eye view feature map, the application realizes fusion of unaligned data collected by two kinds of sensors, obtains a fusion feature map with a complete visual angle range and depth information, and effectively improves learning and training effects.

Description

Technical Field

[0001] This application relates to the field of image fusion technology, and in particular to a fusion method, a 3D target detection method, an in-vehicle device, and a storage medium. Background Technology

[0002] Radar sensors and image sensors are two complementary sensors used for 3D perception in the field of autonomous driving. Radar sensors provide accurate point cloud data for spatial and geometric modeling, while image sensors acquire image data containing rich texture and color information, but lack depth for 3D localization.

[0003] Currently, some L2 and L2+ level sensor architectures typically consist of surround-view cameras and LiDAR covering part of the field of view. However, due to the lack of depth information in the image data, 3D detection without LiDAR coverage can lead to inaccurate bounding box position predictions. Summary of the Invention

[0004] This application provides at least one fusion method, a 3D target detection method, an in-vehicle device, and a storage medium.

[0005] The first aspect of this application provides a fusion method, the fusion method comprising: acquiring point clouds and images for describing a predetermined scene;

[0006] The point cloud is subjected to feature extraction processing to obtain a top-down feature map of the point cloud;

[0007] The image is processed by feature extraction using a trained, pre-defined deep learning model to obtain an overhead feature map.

[0008] Align the point cloud top-view feature map with the image top-view feature map to obtain the point cloud target feature map;

[0009] The point cloud target feature map and the image overhead feature map are input into a trained fusion deep learning model to obtain a fusion feature map for describing the predetermined scene.

[0010] The trained fusion deep learning model includes merging and convolution operations. The point cloud target feature map and the image overhead view feature map are input into the trained fusion deep learning model to obtain a fusion feature map for describing the predetermined scene. This includes performing a channel merging operation on each point cloud overhead view feature in the point cloud target feature map and the corresponding image overhead view feature in the image overhead view feature map to obtain a merging result.

[0011] A convolution operation is performed on the merged result to obtain the fused feature map.

[0012] The similarity loss between the fused features in the fused feature map and the corresponding image overhead features in the image overhead feature map is used to train the preset deep learning model to obtain the trained preset deep learning model.

[0013] The process of aligning the point cloud top-view feature map with the image top-view feature map to obtain the point cloud target feature map includes: performing a range alignment operation on the point cloud top-view feature map and the image top-view feature map to obtain the point cloud intermediate feature map.

[0014] Based on the resolution of the top-view feature map of the image, a mapping operation is performed on the intermediate feature map of the point cloud to obtain the target feature map of the point cloud.

[0015] The process of aligning the point cloud top-view feature map with the image top-view feature map to obtain an intermediate feature map of the point cloud includes: obtaining the alignment range based on the range of the image top-view feature map and the resolution of the point cloud top-view feature map.

[0016] Based on the alignment range, the point cloud overhead feature in the point cloud overhead feature map is filled to obtain the point cloud intermediate feature map.

[0017] The process of extracting features from the point cloud to obtain a top-down feature map of the point cloud includes: performing a voxelization operation on the point cloud to obtain initial voxelized features.

[0018] Using a preset stack of sparse convolutional networks, feature extraction is performed on the initial voxelized features to obtain the point cloud overhead feature map.

[0019] The process of extracting features from the image using a trained deep learning model to obtain an image top-down feature map includes: obtaining a top-down feature map based on the range and target resolution of the image, wherein the top-down feature map is represented by a grid.

[0020] For each grid in the preset overhead feature map, its 3D position is obtained, and it is projected onto the feature map of a preset number of scales corresponding to the panoramic view based on the 3D position. The feature of each scale is interpolated at the projection position to obtain the preset number of features corresponding to the grid.

[0021] The preset number of features are input into the trained preset deep learning model to obtain the image top-down feature map.

[0022] A second aspect of this application provides a 3D object detection method, comprising: acquiring a fused feature map of point cloud and image used to describe a predetermined scene;

[0023] A 3D detection head is used to detect the fused feature map to obtain the 3D detection box of the corresponding 3D target;

[0024] The fused feature map is obtained using the fusion method described in the first aspect above.

[0025] A third aspect of this application provides an in-vehicle device including a memory and a processor coupled to each other, the processor being configured to execute program instructions stored in the memory to implement the fusion method in the first aspect and the 3D target detection method in the second aspect.

[0026] The fourth aspect of this application provides a non-volatile computer-readable storage medium for storing a computer program, which, when executed by a processor, is used to implement the fusion method in the first aspect and the 3D target detection method in the second aspect.

[0027] The aforementioned scheme extracts features from point clouds and images separately to obtain point cloud overhead feature maps and image overhead feature maps. These are then aligned to obtain a point cloud target feature map. Finally, the point cloud target feature map and the image overhead feature map are input into a trained fusion deep learning model to obtain a fused feature map describing a predetermined scene. This application, by aligning the point cloud overhead feature map and the image overhead feature map, achieves the fusion of misaligned data collected by two sensors, resulting in a fused feature map with complete view range and depth information, effectively improving the learning and training effect.

[0028] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0030] Figure 1 This is a flowchart illustrating an embodiment of the fusion method of this application;

[0031] Figure 2 This is a flowchart illustrating an embodiment of the 3D target detection method of this application;

[0032] Figure 3 This is a schematic diagram of the structure of an embodiment of the vehicle-mounted device of this application;

[0033] Figure 4 This is a schematic diagram of the structure of an embodiment of the non-volatile computer-readable storage medium of this application. Detailed Implementation

[0034] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the application. Similarly, the following embodiments are only some, not all, embodiments of the present application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present application.

[0035] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0036] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, "many" in this document means two or more. Additionally, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. Furthermore, the terms "first," "second," and "third" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0037] As mentioned above, some L2 and L2+ level sensor architectures typically consist of surround-view cameras and LiDAR covering part of the field of view. However, due to the lack of depth information in the image data, 3D detection without LiDAR coverage can lead to inaccurate bounding box position predictions.

[0038] Therefore, this application provides a fusion method, a 3D target detection method, an in-vehicle device, and a storage medium.

[0039] Please see Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the fusion method of this application. It should be noted that if substantially the same result is achieved, the method of this application does not necessarily require further elaboration. Figure 1The illustrated process sequence is limited. This method can be applied to in-vehicle devices with computing and other functions. These devices can execute this method by receiving information collected by sensor devices, such as millimeter-wave radar, lidar, or cameras equipped in autonomous vehicles. During the autonomous vehicle's operation, the sensor devices perceive the dynamic real-world scene surrounding the vehicle, including roads, traffic lights, pedestrians, surrounding vehicles, trees, streetlights, buildings, etc. Figure 1 As shown, the fusion method includes the following steps:

[0040] S11. Obtain point cloud and images used to describe the predetermined scene.

[0041] Image sensors can be used to collect road image data, while radar sensors can be used to collect point cloud data. For example, image sensors and radar sensors can be mounted on a mobile device. This mobile device can be an automated mobile device, such as a robot or an autonomous vehicle.

[0042] In some embodiments, the image sensor may be a camera, and the radar sensor may be a lidar sensor, such as a mechanical lidar.

[0043] In one application scenario, an autonomous vehicle travels on a road. Image sensors mounted on the vehicle acquire images of the surrounding environment, and radar sensors obtain corresponding point clouds. This allows for the creation of point clouds and images that describe a predetermined scene. The predetermined scene refers to the environment surrounding the autonomous vehicle when it is in a pre-defined position.

[0044] The point cloud acquired by the radar sensor is a point cloud covering a portion of the field of view, while the image acquired by the image sensor is a panoramic image covering the entire field of view.

[0045] S12. Perform feature extraction processing on the point cloud to obtain an overhead feature map of the point cloud.

[0046] The radar sensor rotates at a constant speed while the autonomous vehicle is moving, collecting point clouds around the vehicle during this rotation to obtain multiple sets of point clouds representing partial viewpoints around the vehicle. Feature extraction is then performed on the point clouds describing the predetermined scene to obtain an overhead feature map of the point clouds.

[0047] Taking the example of a radar sensor rotating once and collecting point clouds four times, the radar sensor obtains four sets of point clouds covering a portion of the field of view in one rotation. "Overhead view" can be understood as looking down from a high place. Feature extraction from these four sets of point clouds yields an overhead feature map of the point clouds.

[0048] Feature extraction of point clouds can be performed using a pre-defined learning model. This pre-defined learning model can be a trained learning model, such as a pre-defined convolutional network, or other methods, which are not limited in this application.

[0049] S13. The image is processed by feature extraction using a pre-trained deep learning model to obtain an overhead feature map of the image.

[0050] The number of image sensors can be set to multiple. During the autonomous vehicle's movement, multiple image sensors acquire images of the surrounding environment to obtain a surround-view image of the vehicle. The number of image sensors can be set to six. In other embodiments, the number of image sensors can also be set to other achievable values, depending on the actual usage requirements, without specific limitations. Feature extraction is performed on the acquired images using a trained pre-defined deep learning model to obtain an image top-down feature map.

[0051] Taking a scenario with six image sensors as an example, these six sensors are positioned around the autonomous vehicle, capturing six images at a time. These six images form a surround view image centered on the autonomous vehicle. A pre-trained deep learning model is used to extract features from these six images, resulting in a top-down feature map centered on the vehicle.

[0052] S14. Align the point cloud top view feature map with the image top view feature map to obtain the point cloud target feature map.

[0053] Since the point cloud acquired by the radar sensor is a point cloud within a partial field of view, it is necessary to align the point cloud top-down feature map with the image top-down feature map to obtain a point cloud target feature map with the same scale as the image top-down feature map.

[0054] S15. Input the point cloud target feature map and the image top-down feature map into the trained fusion deep learning model to obtain a fusion feature map used to describe the predetermined scene.

[0055] The trained fusion deep learning model can be obtained through learning, for example, by inputting image data into the fusion deep learning model and training it to obtain a trained fusion deep learning model that meets preset conditions. The point cloud target feature map obtained through alignment processing and the image top-down feature map are input into the trained fusion deep learning model. The trained fusion deep learning model performs fusion processing on the point cloud target feature map and the image top-down feature map to obtain a fusion feature map used to describe the predetermined scene.

[0056] The aforementioned scheme extracts features from point clouds and images separately to obtain point cloud overhead feature maps and image overhead feature maps. These are then aligned to obtain a point cloud target feature map. Finally, the point cloud target feature map and the image overhead feature map are input into a trained fusion deep learning model to obtain a fused feature map describing a predetermined scene. This application, by aligning the point cloud overhead feature map and the image overhead feature map, achieves the fusion of misaligned data collected by two sensors, resulting in a fused feature map with complete view range and depth information, effectively improving the learning and training effect.

[0057] As described above, inputting the point cloud target feature map and the image overhead view feature map into a trained fusion deep learning model can yield a fusion feature map used to describe a predetermined scene. In one embodiment of this application, the trained fusion deep learning model includes merging operations and convolution operations; inputting the point cloud target feature map and the image overhead view feature map into the trained fusion deep learning model to obtain a fusion feature map used to describe a predetermined scene includes: performing a channel merging operation on each point cloud overhead view feature in the point cloud target feature map and the corresponding image overhead view feature in the image overhead view feature map to obtain a merging result; and performing a convolution operation on the merging result to obtain the fusion feature map.

[0058] The merging operation combines features from different images to obtain a merged result. This can be achieved through the `concat` operation or other feasible methods; no specific limitation is made. The convolution operation performs computations on an image using Convolutional Neural Networks (CNNs) to obtain a convolutional image result.

[0059] Every image can be viewed as a matrix, where each element is a pixel. The image matrix can also be understood as the image's resolution. For example, an image of size H×W can be represented by an H×W matrix, where the value of each element represents the brightness of the pixel at that location. Generally, a larger pixel value indicates a brighter pixel.

[0060] In an image, each primary color represents a channel. A color image contains three primary colors: red (R), green (G), and blue (B), thus having three channels. A grayscale image has one channel. Additionally, there are four-channel images, which consist of the R, G, and B channels plus an alpha channel. The alpha channel is used to represent transparency.

[0061] The trained fusion deep learning model includes merging and convolution operations. The point cloud target feature map contains multiple point cloud target features, and the image overhead view feature map contains multiple image overhead view features. A channel merging operation is performed on each point cloud target feature in the point cloud target feature map and the corresponding image overhead view feature in the image overhead view feature map. For example, the `concat` operation merges the point cloud target features and the image overhead view features through channels to obtain the merged result. The merged result is an image of the same size as both the point cloud target feature map and the image overhead view feature map.

[0062] A convolution operation is performed on the merged result. For example, a convolution operation with a kernel of 3 is performed on the merged result to obtain the fused features in the fused feature map. In other embodiments, the convolution kernel can also be set to other numbers, such as 5, or other achievable values, depending on the actual usage requirements, without specific limitations.

[0063] As described above, a trained, pre-defined deep learning model is used to extract features from an image to obtain an image top-down feature map. In one embodiment of this application, the similarity loss between the fused features in the fused feature map and the corresponding image top-down features in the image top-down feature map is used to train the pre-defined deep learning model to obtain a trained pre-defined deep learning model.

[0064] The similarity loss between the fused features in the fused feature map and the corresponding image overhead features in the image overhead feature map is used to train the preset deep learning model. That is, the similarity loss between the fused features in the fused feature map and the corresponding image overhead features in the image overhead feature map is used to train the deep learning model for extracting image features.

[0065] Similarity can be understood as the similarity between two images. The similarity loss is obtained between the fused features in the fused feature map and the corresponding image overhead features in the image overhead feature map. That is, a loss function is used to calculate the similarity loss between the fused features and the corresponding image overhead features to obtain the corresponding similarity loss. This allows the fused feature map to act as the teacher, and the image overhead features in the image overhead feature map (acting as the student) to perform knowledge distillation training, resulting in a trained pre-defined deep learning model. Notably, the similarity loss is not calculated for locations in the fused feature map that are not covered by point clouds.

[0066] For example, the loss function can be the L1 loss function, also known as MAE (mean absolute error), which is the absolute value of the difference between the predicted value and the true value. In other embodiments, other feasible loss functions can also be used, depending on the actual application requirements, without specific limitations.

[0067] Taking a preset condition of greater than 0.6 and less than or equal to 1 as an example, this means that a calculated similarity loss greater than 0.6 and less than or equal to 1 satisfies the preset condition, while a calculated similarity loss less than or equal to 0.6, or greater than 1, does not satisfy the preset condition. For example, if the similarity loss between the fused feature map and the image overhead feature map is 0.8, which is greater than 0.6 and less than 1, satisfying the preset condition, then the preset deep learning model meets the preset condition, resulting in a trained preset deep learning model. If the similarity loss between the fused feature map and the image overhead feature map is 0.3, which is less than 0.6, not satisfying the preset condition, then the network parameters of the preset deep learning model are adjusted. The adjusted preset deep learning model is used to extract features from the image to obtain a new image overhead feature map. The similarity loss between the fused feature map and the new image overhead feature map is recalculated until the calculated similarity loss satisfies the preset condition, thus obtaining a trained preset deep learning model.

[0068] As described above, aligning the point cloud top-down feature map with the image top-down feature map yields the point cloud target feature map. In one embodiment of this application, aligning the point cloud top-down feature map with the image top-down feature map to obtain the point cloud target feature map includes: performing a range alignment operation on the point cloud top-down feature map and the image top-down feature map to obtain an intermediate feature map of the point cloud; and performing a mapping operation on the intermediate feature map of the point cloud according to the resolution of the image top-down feature map to obtain the point cloud target feature map.

[0069] A range alignment operation is performed on the point cloud overhead feature map and the image overhead feature map to obtain an intermediate point cloud feature map. It can be understood that range represents the sensing range; the range of the point cloud overhead feature map represents the sensing range of the radar sensor, and the range of the image overhead feature map represents the sensing range of the image sensor. Therefore, the range alignment operation is performed on the point cloud overhead feature map and the image overhead feature map to obtain the intermediate point cloud feature map.

[0070] Based on the resolution of the top-down feature map of the image, a mapping operation is performed on the intermediate feature map of the point cloud to obtain the target feature map of the point cloud. The resolution can be the size of the matrix of the top-down feature map of the image. The mapping operation is performed on the intermediate feature map of the point cloud according to the size of the matrix of the top-down feature map of the image, thereby obtaining the target feature map of the point cloud.

[0071] Optionally, the size of the intermediate feature map matrix of the point cloud may differ from the size of the top-view feature map matrix of the image. Mapping operations can be performed on the intermediate feature map of the point cloud using upsampling or downsampling. Specifically, if upsampling is used, the intermediate feature map is enlarged by a factor of S1, resulting in a point cloud target feature map matrix with the same size as the top-view feature map matrix. If downsampling is used, the intermediate feature map is reduced by a factor of S2, resulting in a point cloud target feature map matrix with the same size as the top-view feature map matrix. In other embodiments, other feasible mapping methods can also be used, selected according to actual usage requirements, without specific limitations.

[0072] For example, assuming the resolution of the top-down feature map of the image is 5×5 and the resolution of the intermediate feature map of the point cloud is 8×8, a downsampling operation is used to map the intermediate feature map of the point cloud, that is, to map the intermediate feature map of the point cloud by a factor of 1.6, thereby obtaining a point cloud target feature map with a resolution of 5×5. This ensures that the size of the obtained point cloud target feature map matrix is ​​the same as the size of the top-down feature map matrix of the image.

[0073] For example, assuming the resolution of the top-view feature map of the image is 8×8 and the resolution of the intermediate feature map of the point cloud is 5×5, an upsampling operation is used to map the intermediate feature map of the point cloud, that is, to enlarge the intermediate feature map of the point cloud by a factor of 1.6. On the basis of the matrix of the intermediate feature map of the point cloud, new elements can be inserted between the grids of the matrix using a suitable interpolation algorithm, such as interpolation methods, or other feasible methods, without specific limitations, so as to obtain a point cloud target feature map with a resolution of 8×8. Thus, the size of the obtained point cloud target feature map matrix is ​​the same as the size of the top-view feature map matrix of the image.

[0074] As described above, performing a range alignment operation on the point cloud overhead feature map and the image overhead feature map can obtain an intermediate point cloud feature map. In one embodiment of this application, performing a range alignment operation on the point cloud overhead feature map and the image overhead feature map to obtain an intermediate point cloud feature map includes: obtaining an alignment range based on the range of the image overhead feature map and the resolution of the point cloud overhead feature map; and performing a filling operation on the point cloud overhead features in the point cloud overhead feature map according to the alignment range, thereby obtaining the intermediate point cloud feature map.

[0075] The alignment range is obtained by considering the extent of the image's top-down feature map and the resolution of the point cloud's top-down feature map. It can be understood that the extent of the image's top-down feature map corresponds to the sensor's receptive range, and the resolution of the point cloud's top-down feature map corresponds to the size of the point cloud's top-down feature map matrix. The alignment range can be represented by a matrix, indicating the required size of the point cloud's top-down feature map matrix when performing alignment operations.

[0076] Based on the alignment range, the top-down features of the point cloud are filled in the top-down feature map to obtain the intermediate feature map of the point cloud. In essence, after determining the required alignment range for the top-down feature map, the top-down features are filled. For example, according to the size of the alignment range matrix, the top-down features are filled to fill in any empty spaces, resulting in an intermediate feature map of the point cloud with the same size as the alignment range matrix. The filling operation can use zero-filling or other feasible methods, chosen according to actual needs, without specific limitations.

[0077] For example, assuming the image sensor has a sensing range of 5 meters and the point cloud top-down feature map has a resolution of 15×15, the resulting alignment range is 20×20. Then, zeros are added to the point cloud top-down feature map to obtain a point cloud intermediate feature map with a matrix size of 20×20.

[0078] As described above, feature extraction processing is performed on the point cloud to obtain a top-down feature map of the point cloud. In one embodiment of this application, feature extraction processing of the point cloud to obtain a top-down feature map of the point cloud includes: performing a voxelization operation on the point cloud to obtain initial voxelized features; and using a preset number of stacked sparse convolutional networks to extract features from the initial voxelized features to obtain a top-down feature map of the point cloud.

[0079] Voxelization can expand each point in the collected point cloud into a three-dimensional cubic unit. Voxelization of the point cloud can more accurately, vividly and realistically represent the scanned object, and thus better display the objects in the environment around the autonomous vehicle.

[0080] A voxelization operation is performed on the point cloud to obtain initial voxelized features. A predetermined number of stacked sparse convolutional networks are then used to extract features from these initial voxelized features, resulting in a top-down feature map of the point cloud. The use of sparse convolutional networks to compute and extract features from the initial voxelized features improves computational efficiency.

[0081] As described above, a trained preset deep learning model is used to perform feature extraction processing on the image to obtain an image overhead feature map. In one embodiment of this application, performing feature extraction processing on the image using a trained preset deep learning model to obtain an image overhead feature map includes: obtaining a preset overhead feature map based on the image's range and target resolution, wherein the preset overhead feature map is represented by a grid; for each grid in the preset overhead feature map, obtaining its 3D position, and projecting it onto a preset number of feature maps at different scales corresponding to the panoramic view based on the 3D position, and interpolating the features at each scale at the projection position to obtain a preset number of features corresponding to the grid; and inputting the preset number of features into the trained preset deep learning model to obtain the image overhead feature map.

[0082] Taking a scenario with six image sensors as an example, six images can be acquired at a time, forming a surround-view image centered on the autonomous vehicle. The image range can be determined by the sensor's sensing range, and the resolution can be determined by the size of the image matrix. Based on the sensor's sensing range and the image matrix size, a preset overhead view feature map is obtained. This preset overhead view feature map is represented by a grid, which is essentially a matrix.

[0083] The preset overhead view feature map can be divided into multiple grids. For each grid location, the 3D position of the preset overhead view feature map is calculated. Based on the 3D position and the intrinsic and extrinsic parameters of the radar and image sensors, the portion of the preset overhead view feature map corresponding to each grid location is projected onto a preset number of feature maps of different scales in the surrounding image. Each image in the surrounding image includes a preset number of feature maps, meaning each image contains a preset number of feature maps, and the scales of each feature map are different.

[0084] At the projection location, the features of the feature map at each scale are interpolated. For example, bilinear interpolation or other interpolation methods that can be implemented can be used without specific limitations, thereby obtaining a preset number of features corresponding to each grid location.

[0085] The trained pre-defined deep learning model includes merging and convolution operations. Merging combines different features to obtain a merged result; for example, it can be achieved through the `concat` operation or other feasible methods, without specific limitations. Convolution operations perform Convolutional Neural Network (CNN) calculations on the image to obtain a convolutional image result. A predetermined number of features are input into the trained pre-defined deep learning model, and a channel merging operation is performed on these features, for example, through the `concat` operation, to obtain a third merged result. A convolution operation is then performed on this third merged result; for example, a convolution operation with a kernel of 3 is performed on the third merged result to obtain the image's top-down feature map. In other embodiments, the convolution kernel can also be set to other numbers, such as 5, or other feasible values, depending on the actual usage requirements, without specific limitations.

[0086] Figure 2 This is a flowchart illustrating an embodiment of the 3D target detection method of this application. Please refer to [link / reference]. Figure 2 The 3D target detection method includes the following steps:

[0087] S21. Obtain a fused feature map of point cloud and image used to describe a predetermined scene; wherein the fused feature map is obtained using the fusion method in the above embodiments.

[0088] A fused feature map of point cloud and image used to describe a predetermined scene is obtained, wherein the fused feature map can be obtained using the fusion method described in the above embodiments. For example, point cloud and image used to describe a predetermined scene are obtained; feature extraction processing is performed on the point cloud to obtain a point cloud top-down feature map; feature extraction processing is performed on the image using a trained preset deep learning model to obtain an image top-down feature map; the point cloud top-down feature map and the image top-down feature map are aligned to obtain a point cloud target feature map; the point cloud target feature map and the image top-down feature map are input into a trained fusion deep learning model to obtain a fused feature map used to describe the predetermined scene.

[0089] S22. Use a 3D detection head to detect the fused feature map to obtain the 3D detection box of the corresponding 3D target;

[0090] A 3D detection head is used to detect the fused feature map to obtain the corresponding 3D detection boxes for the 3D targets. The fused features in the fused feature map are supervised to output the 3D detection boxes for the 3D targets. For example, the fused features in the fused feature map can be input into the 3D detection head. The specific type of 3D detection head can be selected according to actual needs and is not specifically limited. The fused features are supervised using preset 3D detection boxes. If the fused features meet the preset parameter conditions, the corresponding 3D detection boxes are output.

[0091] The resulting 3D detection bounding box can be used not only for the detection of 3D targets, but also for the segmentation of 3D targets and the tracking of 3D targets.

[0092] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0093] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an embodiment of the vehicle-mounted device of this application. The vehicle-mounted device 300 includes a memory 301 and a processor 302 coupled to each other. The processor 302 is used to execute program instructions stored in the memory 301 to implement the steps of the above-described fusion method embodiment. In a specific implementation scenario, the vehicle-mounted device 300 may include, but is not limited to, a microcomputer or a server. In addition, the vehicle-mounted device 300 may also include mobile devices such as laptops and tablets, which are not limited here.

[0094] Specifically, processor 302 controls itself and memory 301 to implement the steps of the above-described fusion method embodiment. Processor 302 can also be called a CPU (Central Processing Unit), and may be an integrated circuit chip with signal processing capabilities. Processor 302 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 302 can be implemented using integrated circuit chips.

[0095] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an embodiment of the non-volatile computer-readable storage medium of this application. The computer-readable storage medium 500 is used to store a computer program 501, which, when executed by the processor 302, is used to implement the steps in the above-described fusion method embodiment.

[0096] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0097] In the several embodiments provided in this application, it should be understood that the disclosed methods and related devices can be implemented in other ways. For example, the related device implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication disconnection shown or discussed may be indirect coupling or communication disconnection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0098] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0099] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method of fusing, characterized by, The fusion method includes: Acquire point clouds and images to describe the predetermined scene; The point cloud is subjected to feature extraction processing to obtain a top-down feature map of the point cloud; The image is processed by feature extraction using a trained, pre-defined deep learning model to obtain an overhead feature map. The point cloud top view feature map and the image top view feature map are aligned to obtain the point cloud intermediate feature map. Based on the resolution of the top-view image, a mapping operation is performed on the intermediate feature map of the point cloud to obtain the target feature map of the point cloud; The point cloud target feature map and the image top view feature map are input into a trained fusion deep learning model, which includes merging and convolution operations, to obtain a fusion feature map for describing the predetermined scene. The step of aligning the point cloud top-view feature map with the image top-view feature map to obtain an intermediate point cloud feature map includes: The alignment range is obtained based on the range of the image overhead feature map and the resolution of the point cloud overhead feature map. The range of the image overhead feature map is the sensing range of the image sensor, and the resolution of the point cloud overhead feature map is the matrix size of the point cloud overhead feature map. Based on the alignment range, the point cloud overhead feature in the point cloud overhead feature map is filled to obtain the point cloud intermediate feature map.

2. The fusion method of claim 1, wherein, The step involves inputting the point cloud target feature map and the image top-down feature map into a trained fusion deep learning model. The trained fusion deep learning model includes merging and convolution operations to obtain a fusion feature map describing the predetermined scene, including: Each point cloud top view feature in the point cloud target feature map is combined with the corresponding image top view feature in the image top view feature map through a channel merging operation to obtain the merging result; A convolution operation is performed on the merged result to obtain the fused feature map.

3. The fusion method of claim 1, wherein The similarity loss between the fused features in the fused feature map and the corresponding image overhead features in the image overhead feature map is used to train the preset deep learning model to obtain the trained preset deep learning model.

4. The fusion method according to any one of claims 1 to 3, characterized in that, The point cloud is subjected to feature extraction processing to obtain a top-down feature map of the point cloud, including: Voxelization is performed on the point cloud to obtain initial voxelized features; Using a preset stack of sparse convolutional networks, feature extraction is performed on the initial voxelized features to obtain the point cloud overhead feature map.

5. The fusion method according to any one of claims 1 to 3, characterized in that, The image is processed by a trained, pre-defined deep learning model to extract features, thereby obtaining an image top-down feature map, including: Based on the range and target resolution of the image, a preset overhead feature map is obtained, wherein the preset overhead feature map is represented by a grid. For each grid in the preset overhead feature map, its 3D position is obtained, and it is projected onto the feature map of a preset number of scales corresponding to the panoramic view based on the 3D position. The feature of each scale is interpolated at the projection position to obtain the preset number of features corresponding to the grid. The preset number of features are input into the trained preset deep learning model to obtain the image top-down feature map.

6. A 3D object detection method, characterized in that, include: Obtain a fused feature map of point cloud and image used to describe a predetermined scene; A 3D detection head is used to detect the fused feature map to obtain the 3D detection box of the corresponding 3D target; The fused feature map is obtained using the fusion method described in any one of claims 1-5.

7. An in-vehicle device characterized by comprising: It includes a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory to implement the fusion method as described in any one of claims 1 to 5 and the 3D target detection method as described in claim 6.

8. A non-transitory computer readable storage medium, comprising: The computer-readable storage medium is used to store a computer program, which, when executed by a processor, is used to implement the fusion method as described in any one of claims 1-5 and the 3D target detection method as described in claim 6.

Images