Target detection method and system based on multi-modal fusion, and vehicle

Abstract

The embodiment of the invention provides a multi-modal fusion-based target detection method and system, and a vehicle. The method comprises the steps of obtaining image data and point cloud data in a to-be-recognized traffic scene graph; performing feature extraction on the image data by adopting an image feature extraction branch network to obtain image feature data; performing feature extraction on the point cloud data by adopting a point cloud feature extraction branch network to obtain point cloud feature data; performing feature fusion on the image feature data and the point cloud feature data based on an attention mechanism to obtain fusion features; and a detection network is adopted to detect the fusion feature to obtain a detection result, and the detection network is used for processing a multi-scale and multi-mode target detection task. According to the method and the device, the technical problems of limited single-mode image information capacity and relatively poor capability of adapting to complex scenes in related technologies are solved.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and more specifically, to a target detection method, system, and vehicle based on multimodal fusion. Background Technology

[0002] In intelligent transportation systems and autonomous driving technologies, object detection is a core component for achieving road safety, traffic management, and vehicle navigation. Static elements in real-world traffic scenarios, such as traffic signs and guardrails, exhibit a high degree of similarity to the local features of vehicles. Furthermore, in situations with dense pedestrian traffic and congested roads, occlusion between objects is very common. Current technologies primarily rely on single-modal data for object detection, which has poor adaptability to these complex scenarios and may result in missed detections or misclassifications.

[0003] There is currently no good solution to the above problems. Summary of the Invention

[0004] This application provides a target detection method, system, and vehicle based on multimodal fusion, to at least solve the technical problems in related technologies where the capacity of single-modal image information is limited and the adaptability to complex scenes is poor.

[0005] According to one aspect of the embodiments of this application, a target detection method based on multimodal fusion is provided, comprising: acquiring image data and point cloud data in a traffic scene map to be identified; extracting features from the image data using an image feature extraction branch network to obtain image feature data; extracting features from the point cloud data using a point cloud feature extraction branch network to obtain point cloud feature data; fusing the image feature data and point cloud feature data based on an attention mechanism to obtain fused features; and detecting the fused features using a detection network to obtain a detection result, wherein the detection network is used to handle multi-scale and multimodal target detection tasks.

[0006] Furthermore, the image feature extraction branch network includes a backbone network and a neck network. The image feature extraction branch network is used to extract features from the image data to obtain image feature data, including: extracting features from the image data through the backbone network to obtain multi-scale feature representations; and adjusting the multi-scale feature representations through the neck network to obtain image feature data.

[0007] Furthermore, the point cloud feature extraction branch network includes a voxelization network, a 3D convolutional network, and a bird's-eye view transformation network. The point cloud feature extraction branch network is used to extract features from the point cloud data to obtain point cloud feature data. This includes: voxelizing the point cloud data through the voxelization network to obtain regularly distributed point cloud data; extracting features from the regularly distributed point cloud data through the 3D convolutional network to obtain voxel features; and converting the voxel features into a 2D feature map through the bird's-eye view transformation network to obtain point cloud feature data.

[0008] Furthermore, the attention mechanism includes a channel attention mechanism and a spatial attention mechanism. Based on the attention mechanism, feature fusion is performed on image feature data and point cloud feature data to obtain fused features. This includes: performing average pooling on image feature data and point cloud feature data through the channel attention mechanism to obtain channel attention weights; performing convolution on image feature data and point cloud feature data through the spatial attention mechanism to obtain spatial attention weights; and determining the fused features through the channel attention weights, spatial attention weights, image feature data, and point cloud feature data.

[0009] Furthermore, the fusion features are determined by channel attention weights, spatial attention weights, image feature data, and point cloud feature data, including: multiplying the image feature data and point cloud feature data channel by channel according to the channel attention weights to obtain a channel-weighted feature map; multiplying the image feature data and point cloud feature data pixel by pixel according to the spatial attention weights to obtain a spatially weighted feature map; and fusing the channel-weighted feature map and the spatially weighted feature map element by element to obtain the fusion features.

[0010] Furthermore, the detection results include: target category, bounding box location, and detection confidence. The detection network is used to detect the fused features to obtain the detection results, including: the detection network is used to detect the fused features to obtain the target category of at least one target, the bounding box location of at least one target, and the detection confidence of at least one target.

[0011] Furthermore, the target detection method based on multimodal fusion also includes: filtering the detection confidence of at least one target according to a confidence threshold; in response to any detection confidence being less than the confidence threshold, removing any target corresponding to any detection confidence less than the confidence threshold; and in response to any detection confidence being greater than or equal to the confidence threshold, marking any target corresponding to any detection confidence greater than or equal to the confidence threshold on the traffic scene map to be identified.

[0012] According to another aspect of the embodiments of this application, a target detection system based on multimodal fusion is also provided, comprising: an acquisition module for acquiring image data and point cloud data in a traffic scene map to be identified; a first extraction module for extracting features from the image data using an image feature extraction branch network to obtain image feature data; a second extraction module for extracting features from the point cloud data using a point cloud feature extraction branch network to obtain point cloud feature data; a fusion module for fusing the image feature data and point cloud feature data based on an attention mechanism to obtain fused features; and a detection module for detecting the fused features using a detection network to obtain a detection result, wherein the detection network is used to handle multi-scale and multimodal target detection tasks.

[0013] According to another aspect of the embodiments of this application, a vehicle is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the methods in various embodiments of this application.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0017] In this embodiment, image data and point cloud data of the traffic scene to be identified are first acquired. Then, an image feature extraction branch network is used to extract features from the image data to obtain image feature data, and a point cloud feature extraction branch network is used to extract features from the point cloud data to obtain point cloud feature data. Then, based on an attention mechanism, the image feature data and point cloud feature data are fused to obtain fused features. Finally, a detection network is used to detect the fused features to obtain detection results. By fusing image feature data and point cloud feature data, information in different modalities can be effectively preserved, background noise and illumination interference can be suppressed, and the model's sensitivity to target details can be enhanced. This achieves the goal of accurate multi-scale target identification in complex traffic scenes, thereby improving the technical effect of target detection accuracy and adaptability to complex scenes. This solves the technical problem in related technologies that the single-modal image information capacity is limited and the ability to adapt to complex scenes is poor. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a flowchart of a target detection method based on multimodal fusion according to an embodiment of this application;

[0020] Figure 2 This is a flowchart illustrating an optional multimodal fusion target detection method according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of an optional multimodal fusion target detection network structure according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of a target detection system based on multimodal fusion according to an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] According to an embodiment of this application, an embodiment of a target detection method based on multimodal fusion is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0026] This method embodiment can be executed in an electronic device or similar computing device that includes memory and a processor. Taking operation on a computer terminal as an example, the computer terminal may include one or more processors (processors may include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), digital signal processing (DSP) chips, microcontroller units (MCUs), field-programmable gate arrays (FPGAs), neural network processors (NPUs), tensor processors (TPUs), artificial intelligence (AI) type processors, etc.) and memory for storing data. Optionally, the computer terminal may also include transmission devices, input / output devices, and display devices for communication functions. Those skilled in the art will understand that the above structural description is merely illustrative and does not limit the structure of the computer terminal. For example, the computer terminal may include more or fewer components than described above, or have a different configuration than described above.

[0027] The memory can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the multimodal fusion-based target detection method in this application embodiment. The processor executes various functional applications and data processing by running the computer program stored in the memory, thereby realizing the aforementioned multimodal fusion-based target detection method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0028] The transmission device is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0029] Display devices can be, for example, touchscreen liquid crystal displays (LCDs) and touch displays (also referred to as "touchscreens" or "touch displays"). The LCD allows users to interact with the user interface of the mobile terminal. In some embodiments, the mobile terminal has a graphical user interface (GUI), which allows users to interact with the GUI through finger contact and / or gestures on a touch-sensitive surface. Optional human-computer interaction functions include: creating web pages, drawing, word processing, creating electronic documents, playing games, video conferencing, instant messaging, sending and receiving emails, call interfaces, playing digital video, playing digital music, and / or web browsing, etc. Executable instructions for performing the above human-computer interaction functions are configured / stored in one or more processor-executable computer program products or readable storage media.

[0030] This embodiment provides a target detection method based on multimodal fusion. Figure 1 This is a flowchart of a target detection method based on multimodal fusion according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps:

[0031] Step S10: Obtain image data and point cloud data from the traffic scene map to be identified;

[0032] In this embodiment, the traffic scene map to be identified can be understood as a scene image of an actual road that needs to be detected, captured by an onboard camera. Exemplarily, the traffic scene map to be identified includes, but is not limited to, a combination of elements such as roads, vehicles, pedestrians, obstacles, and traffic signs.

[0033] Image data can be understood as image information in the Red Green Blue (RGB) color mode obtained from the traffic scene image to be identified.

[0034] Point cloud data can be understood as three-dimensional spatial data acquired by vehicle-mounted LiDAR. For example, vehicle-mounted LiDAR emits laser pulses and receives reflected signals to measure the distance and direction of targets, thereby constructing the three-dimensional coordinate information of objects in the environment, forming a series of discrete spatial points, i.e., point cloud data. Point cloud data can provide precise information about the shape, position, and structure of objects, especially in complex traffic scenarios, supplementing the three-dimensional geometric features that image data cannot provide; however, this is not a limitation.

[0035] Acquiring image data and point cloud data from the traffic scene map to be identified can be understood as acquiring RGB image data from the traffic scene map to be identified based on the vehicle-mounted camera, and acquiring point cloud data from the traffic scene map to be identified based on the vehicle-mounted LiDAR.

[0036] It can be seen that by acquiring image data and point cloud data from the traffic scene map to be identified, the system can gain a more comprehensive understanding of the surrounding environment, especially in situations with poor lighting conditions, object occlusion, or long-distance detection. Compared to relying solely on data from a single modality, this approach offers higher accuracy and reliability.

[0037] Step S12: Use an image feature extraction branch network to extract features from the image data to obtain image feature data;

[0038] In this embodiment, the image feature extraction branch network is used to process image data and extract features that are helpful for target recognition from the input RGB image, such as edges, textures, color patterns, and object shapes, etc., which are not limited here. Exemplarily, the image feature extraction branch network can consist of a backbone network and a neck network of an improved YOLOv12 (You Only Look Once version 12). Specifically, the convolutional layers of A2C2f (the original feature extraction module) are replaced with multi-scale depthwise separable convolutional structures to construct the A2C2f_PKI (Patch-based Kernel Injection) module, which includes a 3×3 small kernel convolution and a set of parallel depthwise separable convolutions (5×5, 7×7, 9×9, and 11×11 convolutional kernels). After extracting spatial features through depthwise convolution, the channel attention mechanism is implemented using the Sigmoid activation function. In the ABlock module of A2C2f, the original channel attention module is connected in parallel with the channel-spatial attention module of PKI to form a dual-path attention structure, which is not limited here.

[0039] Image feature data can be understood as important feature descriptions obtained by the image feature extraction branch network through multi-layer convolution, pooling and activation function operations on the input image, without limitation here.

[0040] Using an image feature extraction branch network to extract features from image data can be understood as preprocessing and extracting features from image data through the image feature extraction branch network, converting the original image into an abstract feature representation, i.e., image feature data, thereby providing key feature information for subsequent target detection.

[0041] As can be seen, by acquiring image feature data, the model can capture the key visual features of targets such as distant vehicles, pedestrians, and traffic signs, thereby achieving efficient and accurate target detection in complex and ever-changing traffic environments.

[0042] Step S14: Use a point cloud feature extraction branch network to extract features from the point cloud data to obtain point cloud feature data;

[0043] In this embodiment, the point cloud feature extraction branch network is used to transform 3D point cloud data into feature representations that can be detected by subsequent network modules. For example, a voxel grid architecture is used to encode the point cloud data into voxels, and feature extraction is performed through 3D convolution; however, this is not a limitation.

[0044] Point cloud feature data is generated by the point cloud feature extraction branch network after processing and analysis. For example, point cloud feature data may include, but is not limited to, statistical data within each voxel unit, such as point cloud density, height variation, etc.

[0045] The point cloud feature extraction branch network is used to extract features from point cloud data. The resulting point cloud feature data can be understood as follows: First, the three-dimensional space is divided into multiple voxel units, and the point cloud data within each voxel is aggregated. For example, the average coordinates, the number of count points, and the surface normal or color information are estimated. Then, the voxelized data is feature encoded in order to extract meaningful features from the point cloud data, such as the shape, size, orientation, and surface texture of objects. There are no restrictions on these features.

[0046] It can be seen that the acquired point cloud feature data can provide richer geometric information, which helps to improve the robustness and accuracy of applications such as autonomous driving systems in complex environmental conditions.

[0047] Step S16: Based on the attention mechanism, feature fusion is performed on image feature data and point cloud feature data to obtain fused features;

[0048] In this embodiment, the attention mechanism is used to perform feature fusion on image feature data and point cloud feature data to ensure that the model can focus on valuable information while suppressing irrelevant or redundant parts. Exemplarily, the attention mechanism includes channel attention mechanism and spatial attention mechanism, which are not limited herein.

[0049] Fusion features refer to the comprehensive feature representation obtained by fusing features between image feature data and point cloud feature data.

[0050] The fusion of image feature data and point cloud feature data based on the attention mechanism can be understood as dynamically weighting and fusing image feature data and point cloud feature data according to the channel attention mechanism and the spatial attention mechanism to obtain the fused features.

[0051] It can be seen that the fused features not only retain the main characteristics of the original data, but also better reflect the performance of the target under different modalities, thereby improving the model's ability to identify targets and its adaptability in complex scenarios.

[0052] Step S18: The detection network is used to detect the fused features to obtain the detection results. The detection network is used to handle multi-scale and multi-modal target detection tasks.

[0053] In this embodiment, the detection network can be understood as a neural network model based on the YOLO algorithm, used to handle multi-scale and multi-modal target detection tasks.

[0054] Using a detection network to detect fused features, the detection results can be understood as identifying, classifying, and locating each target in the multi-scale and multi-modal fused features based on the target detection network, thus obtaining the detection results.

[0055] It can be seen that by using a detection network to detect multi-scale and multi-modal fusion features, the accuracy and efficiency of target detection in traffic scenes are significantly improved, providing high-quality input for subsequent intelligent driving decisions, thereby enhancing the safety and functionality of the entire autonomous driving system.

[0056] Through the above steps, image data and point cloud data of the traffic scene to be identified are first acquired. Then, an image feature extraction branch network is used to extract features from the image data to obtain image feature data, and a point cloud feature extraction branch network is used to extract features from the point cloud data to obtain point cloud feature data. Then, based on an attention mechanism, the image feature data and point cloud feature data are fused to obtain fused features. Finally, a detection network is used to detect the fused features to obtain detection results. By fusing image feature data and point cloud feature data, information in different modalities can be effectively preserved, background noise and illumination interference can be suppressed, and the model's sensitivity to target details can be enhanced. This achieves the goal of accurate multi-scale target identification in complex traffic scenes, thereby improving the technical effect of target detection accuracy and adaptability to complex scenes. It also solves the technical problems of limited image information capacity of a single modality and poor adaptability to complex scenes in related technologies.

[0057] Optionally, the image feature extraction branch network includes a backbone network and a neck network. In step S12, the image feature extraction branch network is used to extract features from the image data to obtain image feature data, including the following steps:

[0058] Step S121: Extract features from image data using a backbone network to obtain multi-scale feature representations;

[0059] Step S122: Adjust the multi-scale feature representation through the neck network to obtain image feature data.

[0060] In this embodiment, the backbone network typically consists of multiple convolutional blocks and residual units to capture image features at different levels, such as low-level edge and texture information, and high-level conceptual features, such as the shape and category of the target object. For example, this application uses a backbone network based on an improved version of YOLOv12, which introduces the A2C2f_PKI module. This module combines multi-scale depthwise separable convolution with a channel-space attention mechanism, aiming to more effectively handle the problem of large target scale variations in images and enhance the capture of texture features. However, this is not a limitation.

[0061] The neck network, located between the backbone network and the head network, processes and adjusts the multi-scale feature representations extracted by the backbone network to better suit the needs of the head network. For example, the neck network typically includes structures such as feature pyramid networks, which can perform feature fusion, upsampling, or downsampling operations to ensure that features at different levels can be effectively utilized by the detection head; however, this is not a limitation.

[0062] Multi-scale feature representation refers to the set of features generated after the backbone network analyzes image data at different levels and scales. For example, multi-scale feature representation includes various scale information from small to large targets, helping the detection network to make accurate judgments when faced with targets of different sizes, distances, and occlusion conditions. In this application, the multi-scale feature representation obtained through the use of the A2C2f_PKI module enables the detection network to identify target objects more flexibly and efficiently; however, this is not a limitation.

[0063] Extracting features from image data using a backbone network to obtain multi-scale feature representations can be understood as using a backbone network to process the input image data, thereby extracting feature representations at different levels and scales. For example, low-level feature maps may capture more detailed features such as edges and textures of the image, while high-level feature maps tend to capture more abstract conceptual features such as the shape, location, and category of the target; this is not a limitation here.

[0064] Adjusting multi-scale feature representations through a neck network to obtain image feature data can be understood as using a neck network to further process and adjust the multi-scale feature representations to adapt to the detection requirements of the head network, thereby obtaining image feature data. For example, image feature data can be obtained by adjusting the multi-scale feature representations based on the neck network through a combination of upsampling or downsampling operations and horizontal connections of the feature maps; this is not limited to specific methods.

[0065] It can be seen that by extracting features from image data through the backbone network to obtain multi-scale feature representations, and then adjusting the multi-scale feature representations based on the neck network to obtain image feature data, not only is the network structure simplified and the amount of computation and latency reduced, but the detection accuracy is also improved.

[0066] Optionally, the point cloud feature extraction branch network includes: a voxelization network, a 3D convolutional network, and a bird's-eye view transformation network. In step S14, the point cloud feature extraction branch network is used to extract features from the point cloud data to obtain point cloud feature data, including the following steps:

[0067] Step S141: Voxelize the point cloud data using a voxelization network to obtain regularly distributed point cloud data.

[0068] Step S142: Extract features from regularly distributed point cloud data using a 3D convolutional network to obtain voxel features;

[0069] Step S143: The voxel features are converted into two-dimensional feature maps through a bird's-eye view conversion network to obtain point cloud feature data.

[0070] In this embodiment, a voxelized network refers to a neural network that uses the concept of voxels to preprocess point cloud data.

[0071] Three-dimensional convolutional networks are deep learning models that perform convolution operations on voxelized data.

[0072] The bird's-eye view transformation network is a network used to map three-dimensional voxel features to a two-dimensional plane.

[0073] Two-dimensional feature maps are projections and feature representations of point cloud data onto a two-dimensional plane.

[0074] Voxelization of point cloud data to obtain regularly distributed point cloud data can be understood as transforming point cloud data into a structured, regular voxel grid using a voxelization network.

[0075] Extracting features from regularly distributed point cloud data using a 3D convolutional network to obtain voxel features can be understood as performing multiple convolution and pooling operations on the voxelized point cloud data based on the 3D convolutional network to obtain voxel features containing spatial information such as shape, position, and size.

[0076] Converting voxel features into two-dimensional feature maps using a bird's-eye view transformation network to obtain point cloud feature data can be understood as projecting voxel features onto a two-dimensional plane composed of x and y coordinates using a bird's-eye view transformation network. The resulting two-dimensional feature map (i.e., point cloud feature data) retains the relative position and shape characteristics of the target in space, while removing redundant information in the height dimension, making it easier to fuse or compare with image feature maps.

[0077] As can be seen, through the above steps, the embodiments of this application can transform complex point cloud data into regular and easy-to-process point cloud feature data (i.e., two-dimensional feature maps), thereby providing key point cloud information input for subsequent multimodal target detection systems.

[0078] Optionally, the attention mechanism includes a channel attention mechanism and a spatial attention mechanism. In step S16, feature fusion is performed on image feature data and point cloud feature data based on the attention mechanism to obtain fused features, including the following steps:

[0079] Step S161: Average pooling is performed on the image feature data and point cloud feature data through the channel attention mechanism to obtain the channel attention weights;

[0080] Step S162: Convolutional processing of image feature data and point cloud feature data is performed using a spatial attention mechanism to obtain spatial attention weights;

[0081] Step S163: Determine the fusion features by using channel attention weights, spatial attention weights, image feature data, and point cloud feature data.

[0082] In this embodiment, the channel attention mechanism is used to process channel information in the feature map, that is, to determine the feature types that are more important to the detection task.

[0083] Spatial attention mechanisms are used to process the spatial dimensions of feature maps, i.e., information along the x and y coordinate axes. Spatial attention mechanisms typically perform convolution operations on feature maps to identify and enhance feature responses near the target region, reducing the influence of background or irrelevant regions.

[0084] Channel attention weights are weight matrices generated by the channel attention mechanism through operations such as global average pooling, which reduce the entire feature map into a vector, and then pass through fully connected layers and activation functions.

[0085] Spatial attention weights are weight matrices generated by the spatial attention mechanism through convolution processing of image feature data and point cloud feature data.

[0086] The channel attention weights are obtained by performing average pooling on image feature data and point cloud feature data through the channel attention mechanism. This can be understood as performing global average pooling on the channel information of image feature data and point cloud feature data based on the channel attention mechanism, and then calculating the importance or influence of each channel, i.e., the channel attention weights.

[0087] The spatial attention weights are obtained by convolving image feature data and point cloud feature data using a spatial attention mechanism. This can be understood as follows: convolving image feature data and point cloud feature data using a spatial attention mechanism identifies and enhances regions in the image feature data and point cloud feature data that are relevant to the detection target, while reducing the response of irrelevant regions. By extracting the spatial correlation in the feature data, the spatial attention weight matrix is ​​obtained.

[0088] Determining the fusion feature by using channel attention weights, spatial attention weights, image feature data, and point cloud feature data can be understood as dynamically adjusting the expression intensity of image feature data and point cloud feature data based on channel attention weights and spatial attention weights, and then concatenating the adjusted image feature data and point cloud feature data to obtain the fusion feature.

[0089] As can be seen, the above steps can not only effectively extract and process data from two different modalities, images and point clouds, but also utilize channel attention and spatial attention mechanisms to obtain highly targeted fusion features, thereby enhancing the model's detection capability and robustness.

[0090] Optionally, in step S163, the fusion features are determined by channel attention weights, spatial attention weights, image feature data, and point cloud feature data, including the following steps:

[0091] Step S1631: Multiply the image feature data and point cloud feature data channel by channel according to the channel attention weight to obtain the channel weighted feature map;

[0092] Step S1632: Multiply the image feature data and point cloud feature data pixel by pixel according to the spatial attention weight to obtain a spatially weighted feature map.

[0093] Step S1633: Perform element-level fusion of the channel-weighted feature map and the spatial-weighted feature map to obtain the fused feature.

[0094] In this embodiment, the channel-weighted feature map is obtained by multiplying each channel of the image feature data or point cloud feature data with its respective channel attention weight.

[0095] Spatial weighted feature maps are obtained by multiplying each channel of image feature data or point cloud feature data with its respective spatial attention weight.

[0096] The channel-weighted feature map can be obtained by multiplying the image feature data and point cloud feature data channel by channel attention weight. This can be understood as modulating the information of each channel of the image feature data and point cloud feature data according to the channel attention weight, strengthening the attention to key information and weakening the attention to non-key information, thereby generating the channel-weighted feature map.

[0097] The spatially weighted feature map can be obtained by multiplying the image feature data and point cloud feature data pixel by pixel based on the spatial attention weight. This can be understood as multiplying the spatial attention weight with the feature value of each pixel in the image feature data and point cloud feature data pixel by pixel, thereby enhancing or suppressing the feature response intensity at different locations, thus generating the spatially weighted feature map.

[0098] Element-level fusion of channel-weighted feature maps and spatial-weighted feature maps to obtain fused features can be understood as weighted summation of the feature values ​​at corresponding pixel positions in the channel-weighted feature maps and spatial-weighted feature maps to generate fused features based on multimodality and multiscale.

[0099] It can be seen that by fusing image feature data and point cloud feature data, the resulting fused features are not only optimized in the channel dimension but also enhanced in the spatial dimension. Therefore, they can more comprehensively and accurately represent the information to be detected, thus improving the detection performance of the target detection system in complex scenes.

[0100] Optionally, the detection results include: target category, bounding box location, and detection confidence. In step S18, a detection network is used to detect the fused features to obtain the detection results, including the following steps:

[0101] Step S181: Detect the fused features using a detection network to obtain the target category of at least one target, the bounding box location of at least one target, and the detection confidence of at least one target.

[0102] In this embodiment, the target category refers to the specific type to which the object identified in the image or scene belongs. For example, the target category may include "car", "pedestrian", "bicycle", "motorcycle", etc., and is not limited here.

[0103] The bounding box position refers to the exact location of the detected target in the image, specifically the coordinates of the four vertices of a rectangle or the coordinates of the center point of the rectangle and its width and height. There are no restrictions here.

[0104] Detection confidence is the level of accuracy of a detection model in predicting the category of an object within a bounding box, and is usually expressed as a probability value between 0 and 1.

[0105] Detecting fused features through a detection network to obtain the target category, bounding box location, and detection confidence of at least one target can be understood as determining the target category, bounding box location, and detection confidence of each target in the image based on the detection network's detection of fused features.

[0106] It can be seen that by detecting the fused features, the target category, bounding box position, and detection confidence of the detected target can be determined, thereby achieving accurate positioning of the detected target and improving the overall detection accuracy and robustness.

[0107] Optionally, the target detection method based on multimodal fusion further includes the following steps:

[0108] Step S182: Filter the detection confidence of at least one target according to the confidence threshold;

[0109] Step S183: In response to any detection confidence level being less than the confidence level threshold, remove the target corresponding to any detection confidence level that is less than the confidence level threshold.

[0110] Step S184: In response to any detection confidence level being greater than or equal to a confidence threshold, the target corresponding to any detection confidence level greater than or equal to the confidence threshold is marked on the traffic scene map to be identified.

[0111] In this embodiment of the application, filtering the detection confidence of at least one target based on the confidence threshold can be understood as using the confidence threshold to filter the detection results, so as to avoid too many false alarms or low-quality detection results from affecting the decision-making process.

[0112] Responding to any detection confidence score being less than the confidence threshold, removing any target corresponding to a detection confidence score less than the confidence threshold can be understood as follows: when the detection confidence score of a target in the target detection results output by the detection network is lower than the preset confidence threshold, the target is regarded as an unreliable or low-quality detection result, and therefore needs to be removed.

[0113] In response to any detection confidence level greater than or equal to a confidence threshold, marking any target with a detection confidence level greater than or equal to the confidence threshold on the traffic scene image to be identified can be understood as treating targets with a detection confidence level greater than or equal to the confidence threshold as reliable detection results, and marking the target on the traffic scene image to be identified, displaying the target's category, bounding box, and confidence information.

[0114] It can be seen that by reasonably setting the confidence threshold, detection results with low confidence can be effectively eliminated, and only targets with high confidence can be retained, thereby improving the practicality of the target detection system while ensuring security.

[0115] Figure 2 This is a flowchart illustrating an optional multimodal fusion target detection method according to an embodiment of this application. Figure 3 This is a schematic diagram of an optional multimodal fusion target detection network structure according to an embodiment of this application, such as... Figure 2 and Figure 3 As shown, the dataset is first preprocessed. Bilinear interpolation is used to uniformly adjust the traffic scene image to be identified to a resolution of 640×640, and the image data is normalized based on the mean and standard deviation. A voxel grid architecture is used to divide the original point cloud data into voxels, with the voxel size set to 0.1m×0.1m×0.1m. Local features such as spatial coordinates and point cloud density are extracted from the point cloud within each voxel.

[0116] Next, the backbone and neck network of YOLOv12 were improved. The backbone network is mainly used to extract basic features from the input image, while the neck network is used to fuse multi-level feature maps through upsampling. For example, deep, abstract features are combined with shallow, detailed features (high resolution) through upsampling to generate new feature maps. The convolutional layers of the original A2C2f were replaced with multi-scale depthwise separable convolutional structures to construct the A2C2f_PKI module. The A2C2f_PKI module contains a 3×3 small kernel convolution and a set of parallel depthwise separable convolutions (5×5, 7×7, 9×9, and 11×11 kernels). After extracting spatial features through depthwise convolution, the channel attention mechanism is implemented using the Sigmoid activation function. In the ABlock module of A2C2f, the original channel attention module is connected in parallel with the channel-spatial attention module of PKI to form a dual-path attention structure. By adding a spatial attention branch, the correlation of spatial features is enhanced by improving local context aggregation. The output features of multi-scale branches are merged by channel concatenation. The multi-scale branch output features are added to the module input, preserving the original feature information. Then, a 1×1 convolution is used to adjust the channel dimensions, and the output features are fused to obtain the final output features. The cv1 and cv2 convolutional layers in A2C2f are replaced with a multi-branch depthwise separable convolutional structure of the PKI module, while retaining the original number of channels and activation functions.

[0117] Then, a voxel mesh architecture is used to perform voxel feature encoding (VFE) on the point cloud data, and feature extraction is performed through 3D convolution. A bird's-eye view (BEV) projection method is used to project the output features onto the BEV space to generate a 2D feature map, avoiding target occlusion issues in the 3D scene. YOLOv12's C3K2 (convolutional structure) module and A2C2f structure are used to extract visual features such as texture and edges from the RGB image through multi-stage downsampling.

[0118] Next, a Spatial-Depth Fusion Module (SDFM) is inserted between the backbone network and the neck network. The channel attention mechanism generates weights after global average pooling of the feature maps via fully connected layers. Spatial attention uses the Softmax activation function to calculate the pixel similarity matrix. The two sets of attention weights are then multiplied element-wise with the original features for feature fusion. A detection network model based on YOLOv12 is established, and a Patch-based Pyramid Attention (PPA) module is introduced to improve traditional convolutions, forming a head network with an attention mechanism. The standard 3×3 convolutions in the head network are replaced with the PPA module, retaining the original number of input and output channels and activation functions. The PPA module includes a multi-branch fusion and attention module. The multi-branch fusion part contains three parallel branches: local convolution (3×3 kernel), global convolution (7×7 kernel), and serial convolution (1×1 + 3×3 kernel), extracting features at different scales and levels through different branches. In the patch awareness module, the distinction between local and global branches is achieved by controlling the patch size parameter. Local convolution is performed with p=2 and global convolution is performed with p=4. The results of the three branches are summed to obtain the fused features.

[0119] After multi-branch feature extraction, adaptive feature enhancement is performed using an attention mechanism. The attention module includes channel attention and spatial attention components. Features are processed sequentially using one-dimensional channel attention maps and two-dimensional spatial attention maps, and then the final output is obtained through activation functions and batch normalization. The non-maximum suppression (NMS) algorithm in the loss function is modified to soft-NMS. Soft-NMS modifies the score decay method of the NMS algorithm. It selects the highest-scoring detection box M from the detection list B with scores, removes it from B, and adds it to the final detection set D. It then compares whether other detection boxes overlap with M, and reduces their classification scores based on the degree of overlap with M.

[0120] Finally, the Gaussian decay function is used: Gaussian decay is applied to the scores of overlapping detection boxes M, where 'a' represents the overlap area ratio and 'k' is the decay coefficient. Images from a pre-set dataset are used for algorithm training and testing, resulting in a trained network model. The network model's performance is evaluated by using the trained weights to detect and identify targets on the test set. The traffic scene image to be identified is input into the trained network model, which detects and identifies targets in the image, outputting the identification results, including target category, location bounding box, and confidence score. Through these steps, this application can accurately identify and warn of distant small targets and occluded targets in complex traffic scenes, minimizing road safety issues.

[0121] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0122] According to an embodiment of this application, an embodiment of a target detection system based on multimodal fusion is provided. It should be noted that this system can be used to execute the above-described target detection method based on multimodal fusion.

[0123] Figure 4 This is a schematic diagram of a target detection system based on multimodal fusion according to an embodiment of this application, as shown below. Figure 4 As shown, the multimodal fusion-based target detection system 400 includes: an acquisition module 401 for acquiring image data and point cloud data from a traffic scene map to be identified; a first extraction module 402 for extracting features from the image data using an image feature extraction branch network to obtain image feature data; a second extraction module 403 for extracting features from the point cloud data using a point cloud feature extraction branch network to obtain point cloud feature data; a fusion module 404 for fusing the image feature data and point cloud feature data based on an attention mechanism to obtain fused features; and a detection module 405 for detecting the fused features using a detection network to obtain detection results, wherein the detection network is used to handle multi-scale and multimodal target detection tasks.

[0124] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0125] Embodiments of this application also provide a vehicle, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods described in various embodiments of this application when it runs.

[0126] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0127] Step S10: Obtain image data and point cloud data from the traffic scene map to be identified;

[0128] Step S12: Use an image feature extraction branch network to extract features from the image data to obtain image feature data;

[0129] Step S14: Use a point cloud feature extraction branch network to extract features from the point cloud data to obtain point cloud feature data;

[0130] Step S16: Based on the attention mechanism, feature fusion is performed on image feature data and point cloud feature data to obtain fused features;

[0131] Step S18: The detection network is used to detect the fused features to obtain the detection results. The detection network is used to handle multi-scale and multi-modal target detection tasks.

[0132] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0133] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0134] Step S10: Obtain image data and point cloud data from the traffic scene map to be identified;

[0135] Step S12: Use an image feature extraction branch network to extract features from the image data to obtain image feature data;

[0136] Step S14: Use a point cloud feature extraction branch network to extract features from the point cloud data to obtain point cloud feature data;

[0137] Step S16: Based on the attention mechanism, feature fusion is performed on image feature data and point cloud feature data to obtain fused features;

[0138] Step S18: The detection network is used to detect the fused features to obtain the detection results. The detection network is used to handle multi-scale and multi-modal target detection tasks.

[0139] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the methods in various embodiments of this application.

[0140] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0141] Step S10: Obtain image data and point cloud data from the traffic scene map to be identified;

[0142] Step S12: Use an image feature extraction branch network to extract features from the image data to obtain image feature data;

[0143] Step S14: Use a point cloud feature extraction branch network to extract features from the point cloud data to obtain point cloud feature data;

[0144] Step S16: Based on the attention mechanism, feature fusion is performed on image feature data and point cloud feature data to obtain fused features;

[0145] Step S18: The detection network is used to detect the fused features to obtain the detection results. The detection network is used to handle multi-scale and multi-modal target detection tasks.

[0146] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0147] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0149] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0150] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0151] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0152] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A target detection method based on multimodal fusion, characterized in that, include: Acquire image data and point cloud data from the traffic scene map to be identified; The image feature data is obtained by using an image feature extraction branch network to extract features from the image data. A point cloud feature extraction branch network is used to extract features from the point cloud data to obtain point cloud feature data; The image feature data and the point cloud feature data are fused based on an attention mechanism to obtain fused features; The fused features are detected using a detection network to obtain detection results, wherein the detection network is used to handle multi-scale and multi-modal target detection tasks.

2. The method according to claim 1, characterized in that, The image feature extraction branch network includes a backbone network and a neck network. The image feature extraction branch network is used to extract features from the image data to obtain image feature data, including: The image data is feature extracted using the backbone network to obtain a multi-scale feature representation. The image feature data is obtained by adjusting the multi-scale feature representation through the neck network.

3. The method according to claim 1, characterized in that, The point cloud feature extraction branch network includes: a voxelization network, a 3D convolutional network, and a bird's-eye view transformation network. The point cloud feature extraction branch network is used to extract features from the point cloud data to obtain point cloud feature data, including: The point cloud data is voxelized using the voxelization network to obtain regularly distributed point cloud data. The three-dimensional convolutional network is used to extract features from the regularly distributed point cloud data to obtain voxel features; The voxel features are converted into two-dimensional feature maps using the bird's-eye view conversion network to obtain the point cloud feature data.

4. The method according to claim 1, characterized in that, The attention mechanism includes a channel attention mechanism and a spatial attention mechanism. The feature fusion of the image feature data and the point cloud feature data based on the attention mechanism to obtain fused features includes: The image feature data and the point cloud feature data are subjected to average pooling processing through the channel attention mechanism to obtain the channel attention weights. The spatial attention mechanism is used to perform convolution processing on the image feature data and the point cloud feature data to obtain spatial attention weights. The fused features are determined by the channel attention weights, the spatial attention weights, the image feature data, and the point cloud feature data.

5. The method according to claim 4, characterized in that, The step of determining the fused feature using the channel attention weight, the spatial attention weight, the image feature data, and the point cloud feature data includes: The image feature data and the point cloud feature data are multiplied channel by channel according to the channel attention weights to obtain a channel-weighted feature map; The image feature data and the point cloud feature data are multiplied pixel by pixel according to the spatial attention weight to obtain a spatially weighted feature map; The channel-weighted feature map and the spatial-weighted feature map are fused element-wise to obtain the fused feature.

6. The method according to claim 1, characterized in that, The detection results include: target category, bounding box location, and detection confidence. The detection network is used to detect the fused features to obtain the detection results, which include: The fused features are detected by the detection network to obtain the target category of at least one target, the bounding box position of the at least one target, and the detection confidence of the at least one target.

7. The method according to claim 6, characterized in that, The method further includes: The detection confidence levels corresponding to the at least one target are filtered according to a confidence threshold; In response to any detection confidence level being less than the confidence threshold, the target corresponding to any detection confidence level less than the confidence threshold is removed; In response to any detection confidence level being greater than or equal to the confidence threshold, the target corresponding to any detection confidence level greater than or equal to the confidence threshold is marked on the traffic scene map to be identified.

8. A target detection system based on multimodal fusion, characterized in that, include: The acquisition module is used to acquire image data and point cloud data from the traffic scene map to be identified; The first extraction module is used to extract features from the image data using an image feature extraction branch network to obtain image feature data. The second extraction module is used to extract features from the point cloud data using a point cloud feature extraction branch network to obtain point cloud feature data. The fusion module is used to perform feature fusion on the image feature data and the point cloud feature data based on an attention mechanism to obtain fused features; The detection module is used to detect the fused features using a detection network to obtain detection results, wherein the detection network is used to handle multi-scale and multi-modal target detection tasks.

9. A vehicle, characterized in that, include: Memory, which stores executable programs; A processor for running the executable program, wherein the executable program, when running on the processor, performs the target detection method based on multimodal fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the multimodal fusion-based target detection method as described in any one of claims 1 to 7 when run on a computer or processor.

Images