Multi-modal target detection method and device, vehicle, storage medium and program product

Abstract

The invention relates to the technical field of target detection, in particular to a multi-modal target detection method and device, a vehicle, a storage medium and a program product, and the method comprises the steps: inputting a multi-view image into an image feature extraction network, and outputting image aerial view features through the image feature extraction network; inputting the point cloud data into a radar feature extraction network, and outputting radar aerial view features by the radar feature extraction network; and fusing the image aerial view features and the radar aerial view features, and generating a target detection result in the target area based on the fused aerial view features. Therefore, the problems of insufficient image and point cloud feature extraction, high calculation complexity, low efficiency, difficulty in engineering deployment and the like in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a multimodal target detection method, device, vehicle, storage medium, and program product. Background Art

[0002] Thanks to the successful development of convolutional neural networks, breakthroughs have been made in the field of 3D object detection, with detection accuracy and efficiency continuously improving. Currently, multi-view Bird's Eye View (BEV) methods have become the mainstream approach. While pure vision algorithms such as BEVdet (Bird-Eye-View Detection, a multi-camera 3D object detection paradigm) have achieved promising results by replacing lidar with low-cost solutions, recovering 3D information from 2D images is an ill-posed problem, and multimodal approaches clearly offer superior performance. Current multimodal object detection methods typically utilize on-board cameras and lidar to capture visual and point cloud data. Neural networks such as ResNet (Residual Network) and Transformer are then used to extract image features and map them to a bird's-eye view. Point cloud features are then extracted using methods such as VoxelNet (Voxel Feature Encoding Network) and SECOND (Sparse Convolutional Neural Network). Ultimately, these image and point cloud features are fused to achieve efficient 3D object detection. However, despite significant performance improvements, these methods still face challenges such as insufficient image and point cloud feature extraction, high computational complexity, low computational efficiency, and difficulties in engineering deployment. Summary of the Invention

[0003] The present application provides a multimodal target detection method, device, vehicle, storage medium and program product to solve the problems of insufficient image and point cloud feature extraction, high computational complexity, low efficiency and difficult engineering deployment in related technologies.

[0004] A first aspect of the present application provides a multimodal target detection method, comprising the following steps: acquiring multi-view images and point cloud data of a target area around a vehicle; inputting the multi-view images into an image feature extraction network, wherein the image feature extraction network includes a visual encoder, a feature fusion module, and a first perspective conversion module, the visual encoder being used to extract a multi-scale feature map of the multi-view image, the feature fusion module refining features in the multi-scale feature map based on a cross-attention mechanism, and fusing the refined features based on a feature pyramid network, the first perspective conversion module being used to perform perspective conversion on the fused features to obtain image bird's-eye view features; inputting the point cloud data into a radar feature extraction network, wherein the radar feature extraction network includes a radar encoder, a two-dimensional convolutional neural network, and a second perspective conversion module, the radar encoder outputting voxelized point cloud data, the two-dimensional convolutional neural network performing feature extraction on the voxelized point cloud data, and the second perspective conversion module performing perspective conversion on the extracted features to obtain radar bird's-eye view features; fusing the image bird's-eye view features and the radar bird's-eye view features, and generating a target detection result in the target area based on the fused bird's-eye view features.

[0005] Optionally, fusing the image bird's-eye view features and the radar bird's-eye view features includes: inputting the image bird's-eye view features and the radar bird's-eye view features into a fusion network, and the fusion network outputting the fused bird's-eye view features.

[0006] Optionally, the fusion network includes multiple convolutional layers, batch normalization layers and activation functions, wherein the multiple convolutional layers are used to perform feature mining on the image bird's-eye view features and the radar bird's-eye view features, and integrate the image bird's-eye view features and the radar bird's-eye view features; the batch normalization layer is used to normalize the integrated bird's-eye view features output by the convolutional layer; and the activation function is used to introduce nonlinear change capabilities.

[0007] Optionally, target detection results in the target area are generated based on the fused bird's-eye view features, including: inputting the fused bird's-eye view features into a feedforward neural network, the feedforward neural network extracting semantic information and spatial features of each candidate target in the fused bird's-eye view features, and outputting category prediction results and three-dimensional bounding box parameters of each candidate target in the target area based on the semantic information and spatial features.

[0008] Optionally, the crisscross attention mechanism is: ; in, Represents the features generated by the multi-scale feature map after the cross attention operation, and Represents the scale feature maps of different layers, represents the linear projection operation, and Represent split and reverse split operations, Represents the convolution operation; The feature pyramid network is: ; in, and Represents the scale feature maps of different layers, represents the upsampling operation, Represents the convolution operation.

[0009] Optionally, the coordinate calculation formula of the voxelized point cloud data is: ; ; ; in, Represent points In the voxel grid coordinate; Indicates that the voxel is The resolution or voxel size in the axial direction; Indicates a round-down operation. 、 、 Indicates the minimum value of the range of 3D point cloud data.

[0010] A second aspect of the present application provides a multimodal target detection device, including: an acquisition module for acquiring multi-view images and point cloud data of a target area around a vehicle; a first input module for inputting the multi-view images into an image feature extraction network, wherein the image feature extraction network includes a visual encoder, a feature fusion module, and a first perspective conversion module, the visual encoder is used to extract a multi-scale feature map of the multi-view image, the feature fusion module refines the features in the multi-scale feature map based on a cross-attention mechanism, and fuses the refined features based on a feature pyramid network, the first perspective conversion module is used to perform perspective conversion on the fused features to obtain image bird's-eye view features; a second input module is used to input the point cloud data into a radar feature extraction network, wherein the radar feature extraction network includes a radar encoder, a two-dimensional convolutional neural network, and a second perspective conversion module, the radar encoder outputs voxelized point cloud data, the two-dimensional convolutional neural network performs feature extraction on the voxelized point cloud data, and the second perspective conversion module performs perspective conversion on the extracted features to obtain radar bird's-eye view features; and a fusion module is used to fuse the image bird's-eye view features and the radar bird's-eye view features, and generate a target detection result in the target area based on the fused bird's-eye view features.

[0011] Optionally, the fusion module is further used to: input the image bird's-eye view features and the radar bird's-eye view features into a fusion network, and the fusion network outputs the fused bird's-eye view features.

[0012] Optionally, the fusion network includes multiple convolutional layers, batch normalization layers and activation functions, wherein the multiple convolutional layers are used to perform feature mining on the image bird's-eye view features and the radar bird's-eye view features, and integrate the image bird's-eye view features and the radar bird's-eye view features; the batch normalization layer is used to normalize the integrated bird's-eye view features output by the convolutional layer; and the activation function is used to introduce nonlinear change capabilities.

[0013] Optionally, the fusion module is further used to: input the fused bird's-eye view features into a feedforward neural network, the feedforward neural network extracts the semantic information and spatial features of each candidate target in the fused bird's-eye view features, and outputs the category prediction results and three-dimensional bounding box parameters of each candidate target in the target area based on the semantic information and spatial features.

[0014] Optionally, the crisscross attention mechanism is: ; in, Represents the features generated by the multi-scale feature map after the cross attention operation, and Represents the scale feature maps of different layers, represents the linear projection operation, and Represent split and reverse split operations, Represents the convolution operation; The feature pyramid network is: ; in, and Represents the scale feature maps of different layers, represents the upsampling operation, Represents the convolution operation.

[0015] Optionally, the coordinate calculation formula of the voxelized point cloud data is: ; ; ; in, Represent points In the voxel grid coordinate; Indicates that the voxel is The resolution or voxel size in the axial direction; Indicates a round-down operation. 、 、 Indicates the minimum value of the range of 3D point cloud data.

[0016] The third aspect of the present application provides a vehicle, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to perform the multi-modal target detection method of the above-mentioned embodiments.

[0017] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program or instructions, and the computer program or instructions are executed by a processor to perform the multi-modal target detection method of the above-mentioned embodiments.

[0018] The fifth aspect of the present application provides a computer program product, comprising a computer program or instructions, and the computer program or instructions are executed to implement the multi-modal target detection method of the above-mentioned embodiments.

[0019] Therefore, the present application has at least the following beneficial effects: The embodiments of the present application can use the cross-attention mechanism and pyramid feature cascade fusion in the image feature extraction network to fully extract the image features, improve the image feature expression ability, and use the two-dimensional convolutional neural network to replace the traditional sparse convolution and three-dimensional convolution in the radar feature extraction network, thereby reducing the computational complexity and resource consumption in point cloud processing, and the two-dimensional convolutional neural network is easier to convert the format, so as to be deployed on different platforms, reduce the deployment difficulty, and generate the target detection result in the target region based on the fused bird's eye view features. Therefore, the technical problems of insufficient image and point cloud feature extraction, high computational complexity, low efficiency, and difficult engineering deployment in the related art are solved.

[0020] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which: Figure 1 A flowchart of the multi-modal target detection method according to the embodiments of the present application is provided. Figure 2 An overall architecture diagram of the multi-modal target detection method according to the embodiments of the present application is provided. Figure 3 An example diagram of the multi-modal target detection device according to the embodiments of the present application is provided. Figure 4 A structural schematic diagram of the vehicle according to the embodiments of the present application is provided. DETAILED DESCRIPTION

[0022] Embodiments of the present application are described below in detail, examples of which are shown in the accompanying drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0023] The multi-modal target detection method, device, vehicle, storage medium and program product of the embodiments of the present application are described below with reference to the accompanying drawings. In view of the above background technology, the current multi-modal target detection method generally captures visual and point cloud data by using vehicle-mounted cameras and laser radars, extracts image features by using neural networks such as ResNet / Transformer, maps to a bird's eye view, extracts point cloud features by using VoxelNet / SECOND, and finally fuses image features and point cloud features to realize efficient 3D target detection. However, although such methods have made significant progress in performance, they still face challenges such as insufficient image and point cloud feature extraction, high computational complexity, low computational efficiency, and difficult engineering deployment. The present application provides a multi-modal target detection method. In this method, a cross-attention mechanism and pyramid feature cascade fusion can be used in the image feature extraction network to fully extract image features and improve the image feature expression capability. In addition, a two-dimensional convolutional neural network is used in the radar feature extraction network to replace the traditional sparse convolution and three-dimensional convolution, which reduces the computational complexity and resource consumption in point cloud processing. The two-dimensional convolutional neural network is easier to convert formats for deployment on different platforms, reducing deployment difficulty. The image bird's eye view features output by the image feature extraction network and the radar bird's eye view features output by the radar feature extraction network are fused to generate target detection results in the target region based on the fused bird's eye view features. Thus, the problems of insufficient image and point cloud feature extraction, high computational complexity, low efficiency, and difficult engineering deployment in related technologies are solved.

[0024] Specifically, Figure 1 A flowchart of a multi-modal target detection method provided by an embodiment of the present application is shown.

[0025] As Figure 1 shown, the multi-modal target detection method includes the following steps: In step S101, multi-view images and point cloud data of a target region around the vehicle are obtained.

[0026] The multi-view images can be obtained by vehicle-mounted cameras and vehicle-mounted cameras, and the point cloud data can be captured by vehicle-mounted laser radars.

[0027] In step S102, the multi-view image is input to an image feature extraction network, wherein the image feature extraction network comprises a visual encoder, a feature fusion module and a first view conversion module, the visual encoder is used to extract a multi-scale feature map of the multi-view image, the feature fusion module refines features in the multi-scale feature map based on a cross attention mechanism and fuses the refined features based on a feature pyramid network, and the first view conversion module is used to perform view conversion on the fused features to obtain image bird's eye view features.

[0028] The first view conversion module can also be referred to as a first BEV conversion module, and the bird's eye view features can be referred to as BEV features.

[0029] It can be understood that the embodiments of the present application can input a multi-view image to an image feature extraction network, the image feature extraction network outputs image bird's eye view features, and the image feature network comprises a visual encoder, a feature fusion module and a first view conversion module, wherein, The visual encoder can be ResNet or Swin Transformer, and the visual encoder is used to extract a multi-scale feature map from top to bottom; The feature fusion module includes a cross attention mechanism and a feature pyramid network, the cross attention mechanism refines the multi-scale feature map, the cross attention mechanism can enhance the interaction between different levels of feature information, and can also capture the global context dependency relationship between high-level semantic features and low-level detail features, and the refined features enter the pyramid network to complete the fusion of multi-scale information from bottom to top; The first view conversion module is used to map the fused features from the camera view to a unified bird's eye view space, which lays a foundation for subsequent fusion of bird's eye view features with point clouds and 3D perception.

[0030] The embodiments of the present application can improve the problems of insufficient feature extraction and imperfect feature expression by using the cross attention mechanism and pyramid feature cascade fusion in the image feature extraction network.

[0031] In the embodiments of the present application, the cross attention mechanism is: ; Wherein, represents the features generated after the cross attention operation on the multi-scale feature map, and represent the scale feature maps of different layers, represents a linear projection operation, and represent split and reverse split operations respectively, represents a convolution operation; The feature pyramid network is: ; wherein, and represent the scale feature maps of different layers, represents an up-sampling operation, represents a convolution operation.

[0032] In step S103, the point cloud data is input to the radar feature extraction network, wherein the radar feature extraction network comprises a radar encoder, a two-dimensional convolutional neural network and a second view conversion module, the radar encoder outputs the voxelized point cloud data, the two-dimensional convolutional neural network extracts features from the voxelized point cloud data, and the second view conversion module converts the extracted features to obtain radar bird's eye view features.

[0033] wherein the two-dimensional convolutional neural network can be a PointNet of PointPillars; It can be understood that the point cloud data can be input to the radar feature extraction network, and the radar feature extraction network outputs radar bird's eye view features, and the radar feature extraction network comprises a radar encoder, a two-dimensional convolutional neural network and a second view conversion module, wherein, The radar encoder is used to voxelize the three-dimensional space according to the set parameters, and to disperse the point cloud into regular pillars; The two-dimensional convolutional neural network is used to extract features from the points in each voxel, generate a fixed-dimensional voxel feature vector, and arrange all pillar features into a sparse pseudo-image according to the spatial position, process the pseudo-image through a standard two-dimensional convolutional neural network, and extract high-level semantic features; The second view conversion module is used to project the information (such as reflection intensity, distance or speed) of the pillars in the height dimension to the corresponding pixel position in the BEV space, realize effective mapping from three-dimensional point cloud to two-dimensional bird's eye view, and generate radar bird's eye view features representing spatial structure and semantic information.

[0034] The embodiments of the present application can use only two-dimensional convolution to process the voxelized point cloud in the point cloud processing process, so that the network structure is easy to implement and debug, the computing efficiency is improved, and multiple open source platforms are supported, which is easy to deploy in engineering.

[0035] In the embodiments of the present application, the coordinate calculation formula of the voxelized point cloud data is: ; ; ; wherein, respectively represent the point in the voxel grid coordinates; represents the resolution or voxel size of the voxel in the axis direction; represents a floor operation, , , represents the minimum value of the range of the three-dimensional spatial point cloud data.

[0036] Specifically, the voxelization of the embodiments of the present application is to discretize the point cloud data in the three-dimensional space into a regular voxel grid. The voxelization process usually includes the following steps: (1) Determine the size of the voxel (voxel resolution): define the resolution of the voxel in each axis (X, Y, Z).

[0037] (2) Determine the range of the voxel grid: that is, the spatial range where the data is located, which is usually determined by the maximum and minimum coordinates of the point cloud.

[0038] (3) Map each point to the voxel it is in: according to the resolution of the voxel and the spatial range of the data, map the coordinates of each point cloud to the corresponding voxel.

[0039] In step S104, the image bird's eye feature and the radar bird's eye feature are fused, and a target detection result in the target region is generated based on the fused bird's eye feature.

[0040] Suppose the range of the three-dimensional spatial point cloud data is , the size of the voxel is , and for each point , the voxel coordinates where it is located are: ; ; ; wherein , respectively, are the coordinates of the point in the voxel grid; is the resolution or voxel size of the voxel in the axis direction; represents a floor operation.

[0041] It can be understood that the embodiments of the present application can fuse the image bird's eye feature and the radar bird's eye feature, and generate a target detection result in the target region based on the fused bird's eye feature.

[0042] In the embodiments of the present application, the image bird's eye feature and the radar bird's eye feature are fused, including: inputting the image bird's eye feature and the radar bird's eye feature into a fusion network, and the fusion network outputs the fused bird's eye feature.

[0043] It can be understood that the image bird's eye view feature and the radar bird's eye view feature can be input into the fusion network, and the fusion network outputs the fused bird's eye view feature.

[0044] In the embodiment of the application, the fusion network comprises a plurality of convolution layers, a batch normalization layer and an activation function, wherein the plurality of convolution layers are used to mine features of the image bird's eye view feature and the radar bird's eye view feature and integrate the image bird's eye view feature and the radar bird's eye view feature; the batch normalization layer is used to normalize the integrated bird's eye view feature output by the convolution layer; and the activation function is used to introduce a nonlinear change capability.

[0045] Specifically, in order to realize efficient integration of geometric structure and semantic appearance information, the image BEV feature and the point cloud BEV feature are fused in a unified space, so as to construct a more powerful and robust 3D perception system. The fusion network is composed of a plurality of convolution layers, a batch normalization layer and an activation function, and is used to deeply mine and integrate multi-modal features. The image BEV and the radar BEV feature are input into the fusion network as input, and finally the fused BEV feature map is output, which is used for subsequent class prediction and target detection tasks.

[0046] In the embodiment of the application, the target detection result in the target region is generated based on the fused bird's eye view feature, comprising: inputting the fused bird's eye view feature into a feed-forward neural network, the feed-forward neural network extracting semantic information and spatial features of each candidate target in the fused bird's eye view feature, and outputting a class prediction result and a three-dimensional bounding box parameter of each candidate target in the target region based on the semantic information and the spatial features.

[0047] The three-dimensional bounding box parameter can include position, size, orientation, etc.

[0048] Specifically, the fused bird's eye view feature can be input into the FFN (Feed-Forward Network, feed-forward neural network), the FFN extracts semantic information and spatial features of each candidate target, outputs corresponding class prediction results and three-dimensional bounding box parameters (such as position, size, orientation, etc.), and finally obtains complete 3D target detection results.

[0049] In summary, the embodiment of the present application adopts a cross-attention mechanism in the image feature extraction process. Unlike the traditional convolutional neural network which can only capture local information, the attention mechanism calculates the relationship weight between each position in the image, performs fine-grained modeling in the global scope of the image, enhances the interaction between different levels of features, and captures a wide range of feature dependencies. A feature pyramid network is used to perform lightweight fusion of multi-scale feature maps, which not only reduces the perceptual bias that may be caused by single-scale features, but also improves the integrity and robustness of semantic expression. The two work together to enhance the perception ability and stability of the model. In the point cloud feature extraction process, a voxelization convolutional neural network based on PointPillars is used to efficiently extract sparse point cloud features. By using a traditional neural network structure to replace the complex calculations of sparse convolution and 3D convolution, the computational complexity and resource consumption of the network are reduced, making the overall structure more concise and efficient, and improving the processing efficiency and accuracy of point cloud and image data. Compared with the sparse convolution method, the method of the present application is easier to implement in engineering, the structure is easier to convert to ONNX format, and is compatible with mainstream deployment platforms such as NVIDIA, has higher deployment flexibility, greatly improves the portability and engineering deployment efficiency of the method, and is convenient for rapid landing and application in actual scenarios.

[0050] The multi-modal target detection method of the present application will be described below through a specific embodiment. The overall architecture is as shown in Figure 2 The method specifically includes: Step 1, image feature extraction.

[0051] The image feature extraction network is composed of a visual encoder, a feature fusion module, and a BEV conversion module. After data augmentation, multi-view RGB images are input into the backbone network (such as ResNet or Swin Transformer) to extract multi-scale feature maps from top to bottom. In order to enhance the information interaction between different levels of features, a cross-attention mechanism is introduced to refine the feature maps and capture the global context dependency relationship between high-level semantic features and low-level detail features. The refined features enter the feature pyramid network to complete the fusion of multi-scale information from bottom to top. The fused image features are mapped from the camera perspective to the unified bird's eye view (BEV) space through the BEV conversion module, laying the foundation for subsequent fusion with point cloud BEV features and 3D perception.

[0052] Step 2, point cloud feature extraction.

[0053] After the data augmentation of the three-dimensional point cloud, the three-dimensional point cloud is input into the radar encoder. First, the three-dimensional space is voxelized according to the set parameters, and the point cloud is discretized into regular columnar bodies. Then, the PointNet is used to extract the features of the points in each voxel, generate a fixed-dimensional voxel feature vector, and arrange all the columnar body features into a sparse pseudo-image according to the spatial position. The pseudo-image is processed by a standard two-dimensional convolutional neural network to extract high-level semantic features. Finally, the information of the columnar body in the height dimension (such as reflection intensity, distance or speed) is projected to the corresponding pixel position in the BEV space, realizing effective mapping from three-dimensional point cloud to two-dimensional bird's eye view, and generating a radar BEV feature map representing the spatial structure and semantic information.

[0054] Step 3, image point cloud feature fusion.

[0055] In order to realize the efficient integration of geometric structure and semantic appearance information, the image BEV feature and the point cloud BEV feature are fused in a unified space, thereby constructing a more powerful and robust 3D perception system. The fusion network is composed of multiple convolutional layers, batch normalization layers and activation functions, which are used to deeply mine and integrate multi-modal features. The image BEV and the radar BEV feature are input into the fusion network as input, and finally the fused BEV feature map is output, which is used for subsequent class prediction and target detection tasks.

[0056] Step 4, 3D target detection.

[0057] The fused feature is then input into the FFN decoder for decoding to extract the semantic information and spatial features of each candidate target, output the corresponding class prediction result and three-dimensional bounding box parameters (such as position, size, orientation, etc.), and finally obtain the complete 3D target detection result.

[0058] In this embodiment, the voxelization neural network based on the traditional convolutional layer of PointPillars is used instead of the neural network structure such as sparse convolution and 3D convolution of SECOND, which reduces the computational complexity and engineering deployment difficulty in laser radar point cloud processing, and uses the cross attention mechanism and pyramid feature cascade fusion in the visual feature extraction network, which improves the problem of insufficient feature extraction and imperfect feature expression. The joint feature extraction and fusion of point cloud and image are carried out at the BEV level, which breaks through the problem of insufficient information fusion in the traditional feature fusion method.

[0059] According to the multi-modal target detection method provided in the embodiments of the present application, the cross attention mechanism and the pyramid feature cascade fusion can be used in the image feature extraction network to fully extract the image features, improve the image feature expression capability, and the two-dimensional convolutional neural network is used in the radar feature extraction network to replace the traditional sparse convolution and three-dimensional convolution, so as to reduce the calculation complexity and resource consumption in the point cloud processing, and the two-dimensional convolutional neural network is more easy to convert the format to facilitate the deployment on different platforms, reduce the deployment difficulty, and the image bird's eye view features output by the image feature extraction network and the radar bird's eye view features output by the radar feature extraction network are fused to generate the target detection result in the target region based on the fused bird's eye view features.

[0060] Secondly, the multi-modal target detection device provided in the embodiments of the present application is described with reference to the accompanying drawings.

[0061] Figure 3 is a block schematic diagram of the multi-modal target detection device in the embodiments of the present application.

[0062] As shown in Figure 3 , the multi-modal target detection device 10 comprises an acquisition module 100, a first input module 200, a second input module 300 and a fusion module 400.

[0063] The acquisition module 100 is configured to acquire multi-view images and point cloud data of a target region around a vehicle; the first input module 200 is configured to input the multi-view images to an image feature extraction network, wherein the image feature extraction network comprises a visual encoder, a feature fusion module and a first view conversion module, the visual encoder is configured to extract multi-scale feature maps of the multi-view images, the feature fusion module is configured to refine features in the multi-scale feature maps based on a cross attention mechanism and fuse the refined features based on a feature pyramid network, and the first view conversion module is configured to perform view conversion on the fused features to obtain image bird's eye view features; the second input module 300 is configured to input the point cloud data to a radar feature extraction network, wherein the radar feature extraction network comprises a radar encoder, a two-dimensional convolutional neural network and a second view conversion module, the radar encoder outputs voxelized point cloud data, the two-dimensional convolutional neural network extracts features from the voxelized point cloud data, and the second view conversion module performs view conversion on the extracted features to obtain radar bird's eye view features; and the fusion module 400 is configured to fuse the image bird's eye view features and the radar bird's eye view features, and generate a target detection result in the target region based on the fused bird's eye view features.

[0064] In the embodiments of the present application, the fusion module 400 is further configured to input the image bird's eye view features and the radar bird's eye view features to a fusion network, and the fusion network outputs the fused bird's eye view features.

[0065] In an embodiment of the present application, the fusion network includes multiple convolutional layers, batch normalization layers and activation functions, wherein the multiple convolutional layers are used to perform feature mining on the image bird's-eye view features and the radar bird's-eye view features, and integrate the image bird's-eye view features and the radar bird's-eye view features; the batch normalization layer is used to normalize the integrated bird's-eye view features output by the convolutional layer; and the activation function is used to introduce nonlinear change capabilities.

[0066] In an embodiment of the present application, the fusion module 400 is further used to: input the fused bird's-eye view features into a feedforward neural network, the feedforward neural network extracts the semantic information and spatial features of each candidate target in the fused bird's-eye view features, and outputs the category prediction results and three-dimensional bounding box parameters of each candidate target in the target area based on the semantic information and spatial features.

[0067] In this embodiment of the present application, the cross attention mechanism is: ; in, Represents the features generated by the multi-scale feature map after the cross attention operation, and Represents the scale feature maps of different layers, represents the linear projection operation, and Represent split and reverse split operations, Represents the convolution operation; The feature pyramid network is: ; in, and Represents the scale feature maps of different layers, represents the upsampling operation, Represents the convolution operation.

[0068] In the embodiment of the present application, the coordinate calculation formula of the voxelized point cloud data is: ; ; ; in, Represent points In the voxel grid coordinate; Indicates that the voxel is The resolution or voxel size in the axial direction; Indicates a round-down operation. 、 、 Indicates the minimum value of the range of 3D point cloud data.

[0069] It should be noted that the foregoing explanation and description of the multi-modal target detection method embodiment also applies to the multi-modal target detection device of this embodiment, which will not be described here.

[0070] The multi-modal target detection device provided by the embodiment of the application can use cross attention mechanism and pyramid feature cascade fusion in the image feature extraction network to fully extract image features, improve the image feature expression capability, and use a two-dimensional convolutional neural network to replace the traditional sparse convolution and three-dimensional convolution in the radar feature extraction network, thereby reducing the calculation complexity and resource consumption in point cloud processing, and the two-dimensional convolutional neural network is easier to convert the format to facilitate deployment on different platforms, thereby reducing the deployment difficulty. The image bird's eye view features output by the image feature extraction network and the radar bird's eye view features output by the radar feature extraction network are fused, and the target detection result in the target region is generated based on the fused bird's eye view features.

[0071] Figure 4 The vehicle provided by the embodiment of the application is shown in the structural schematic diagram. The vehicle can include: The memory 401, the processor 402, and the computer program stored in the memory 401 and executable on the processor 402.

[0072] The processor 402 implements the multi-modal target detection method provided in the above embodiments when executing the program.

[0073] Further, the vehicle further includes: The communication interface 403 is used for communication between the memory 401 and the processor 402.

[0074] The memory 401 is used to store the computer program executable on the processor 402.

[0075] The memory 401 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0076] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 4 Only one thick line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0077] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can complete communication between each other through an internal interface.

[0078] The processor 402 can be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0079] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program or instructions, and the computer program or instructions are executed by a processor to implement the multi-modal target detection method.

[0080] The embodiment of the present application further provides a computer program product, which includes a computer program or instructions, and the computer program or instructions are executed to implement the multi-modal target detection method.

[0081] In the description of the application, reference to "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that a particular feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. The illustrative appearances of the above-mentioned terms in various places in the specification are not necessarily referred to the same embodiment or example. Moreover, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples. Furthermore, in non-contradictory cases, those skilled in the art can combine and combine the features of different embodiments or examples described in the specification and the features of different embodiments or examples.

[0082] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0083] Any process or method descriptions or descriptions of the flow diagrams described herein or otherwise described in this application can be understood as representing the steps of a method implemented by one or N executable instructions, code segments, or portions of code, for implementing custom logic or a process, and that the scope of preferred embodiments of the present application encompasses additional implementation in which the steps, functions, procedures, methods, or routines are performed in an order different from that described, including substantially concurrently or in reverse order, depending upon the functions involved, as would be understood by those skilled in the art.

[0084] It should be understood that portions of the application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, N steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. As in another embodiment implemented with hardware, any one or a combination of the following technologies known in the art can be used: discrete logic circuit with logic gates for implementing logic functions on data signals, application specific integrated circuit with appropriate combination logic gates, programmable gate array, field programmable gate array, etc.

[0085] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment method can be completed by program instructions to the relevant hardware, and the program can be stored in a computer readable storage medium. The program, when executed, includes one or a combination of the steps of the method embodiment.

Claims

1. A multimodal target detection method, characterized in that: The following steps are involved: Acquire multi-view images and point cloud data of the target area around the vehicle; Inputting the multi-view image into an image feature extraction network, wherein the image feature extraction network includes a visual encoder, a feature fusion module, and a first perspective conversion module. The visual encoder is used to extract a multi-scale feature map of the multi-view image. The feature fusion module refines the features in the multi-scale feature map based on a cross-attention mechanism and fuses the refined features based on a feature pyramid network. The first perspective conversion module is used to perform perspective conversion on the fused features to obtain a bird's-eye view feature of the image. Inputting the point cloud data into a radar feature extraction network, wherein the radar feature extraction network includes a radar encoder, a two-dimensional convolutional neural network, and a second perspective conversion module. The radar encoder outputs voxelized point cloud data, the two-dimensional convolutional neural network performs feature extraction on the voxelized point cloud data, and the second perspective conversion module performs perspective conversion on the extracted features to obtain radar bird's-eye view features. The image bird's-eye view feature and the radar bird's-eye view feature are fused, and a target detection result in the target area is generated based on the fused bird's-eye view feature.

2. The multimodal target detection method according to claim 1, wherein: The fusing of the image bird's-eye view feature and the radar bird's-eye view feature includes: The image bird's-eye view features and the radar bird's-eye view features are input into a fusion network, and the fusion network outputs fused bird's-eye view features.

3. The multimodal target detection method according to claim 2, wherein: The fusion network includes multiple convolutional layers, batch normalization layers and activation functions, wherein, The multiple convolutional layers are used to perform feature mining on the image bird's-eye view features and the radar bird's-eye view features, and integrate the image bird's-eye view features and the radar bird's-eye view features; The batch normalization layer is used to normalize the integrated bird's-eye view features output by the convolutional layer; The activation function is used to introduce nonlinear change capabilities.

4. The multimodal target detection method according to claim 1, wherein: Generating the target detection result in the target area based on the fused bird's-eye view features includes: The fused bird's-eye view features are input into a feedforward neural network, which extracts the semantic information and spatial features of each candidate target from the fused bird's-eye view features, and outputs the category prediction results and three-dimensional bounding box parameters of each candidate target in the target area based on the semantic information and spatial features.

5. The multimodal target detection method according to claim 1, wherein: The cross attention mechanism is: ； in, Represents the features generated by the multi-scale feature map after the cross attention operation, and Represents the scale feature maps of different layers, represents the linear projection operation, and Represent split and reverse split operations, Represents the convolution operation; The feature pyramid network is: ； in, and Represents the scale feature maps of different layers, represents the upsampling operation, Represents the convolution operation.

6. The multimodal target detection method according to claim 1, wherein: The coordinate calculation formula of the voxelized point cloud data is: ； ； ； in, Represent points In the voxel grid coordinate; Indicates that the voxel is The resolution or voxel size in the axial direction; Indicates a round-down operation. 、 、 Indicates the minimum value of the range of 3D point cloud data.

7. A multimodal target detection device, characterized in that: include: An acquisition module is used to acquire multi-view images and point cloud data of the target area around the vehicle; a first input module, configured to input the multi-view image into an image feature extraction network, wherein the image feature extraction network includes a visual encoder, a feature fusion module, and a first perspective conversion module; the visual encoder is configured to extract a multi-scale feature map of the multi-view image; the feature fusion module refines features in the multi-scale feature map based on a cross-attention mechanism and fuses the refined features based on a feature pyramid network; and the first perspective conversion module is configured to perform perspective conversion on the fused features to obtain a bird's-eye view feature of the image; a second input module, configured to input the point cloud data into a radar feature extraction network, wherein the radar feature extraction network includes a radar encoder, a two-dimensional convolutional neural network, and a second perspective conversion module; the radar encoder outputs voxelized point cloud data; the two-dimensional convolutional neural network performs feature extraction on the voxelized point cloud data; and the second perspective conversion module performs perspective conversion on the extracted features to obtain radar bird's-eye view features; A fusion module is used to fuse the image bird's-eye view features and the radar bird's-eye view features, and generate a target detection result in the target area based on the fused bird's-eye view features.

8. A vehicle, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimodal target detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: The computer program or instruction is executed by a processor to implement the multimodal target detection method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed, the multimodal target detection method according to any one of claims 1 to 6 is implemented.

Images