Target detection method, system, device and medium based on multi-sensor fusion

Abstract

The application provides a target detection method, system, device and medium based on multi-sensor fusion, belonging to the technical field of computer vision. The method comprises the following steps: acquiring radar point cloud data and multi-view image data of the environment around the vehicle; performing perspective division and feature extraction on the radar point cloud data to obtain multi-view radar point cloud features and multi-view radar point cloud semantic features; combining the multi-view radar point cloud features and a plurality of camera parameter matrices to perform feature extraction and depth prediction on the multi-view image data, thereby obtaining multi-view image semantic features and multi-view image depth information; fusing the multi-view image semantic features and the multi-view radar point cloud semantic features according to the multi-view image depth information, thereby obtaining multi-modal fusion features in the bird's eye view perspective; and performing target detection on the multi-modal fusion features, thereby obtaining a target detection result. The application can more accurately convert multi-view image features to the bird's eye view perspective, thereby improving the reliability of the target detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a target detection method and system based on multi-sensor fusion, a device and a medium. BACKGROUND

[0002] In the automatic driving perception scene, various sensors such as camera devices, millimeter wave radar devices and laser radar devices are mainly relied on to obtain the vehicle surrounding environment information, and then deep learning technology is combined to fuse and analyze various sensor data to further realize target recognition and positioning. On the problem of multi-sensor data fusion, the point cloud data collected by the related radar device itself contains three-dimensional spatial position information, which can be easily converted to the bird's eye view perspective for representation, but the two-dimensional image collected by the camera device lacks depth information, and the LSS algorithm is usually used to predict the depth information corresponding to each pixel contained in the two-dimensional image to convert the two-dimensional image to the bird's eye view perspective for representation. However, the lack of reliable depth information support when applying the LSS algorithm will cause overall deviation in image depth estimation, so that the image features are not accurate enough when converted to the bird's eye view perspective, and finally affect the reliability of the target detection result. SUMMARY

[0003] The main purpose of the present application is to propose a target detection method and system based on multi-sensor fusion, which can predict reliable depth information for multi-view image data to improve the accuracy of multi-view image features when converted to the bird's eye view perspective, and is beneficial to improve the reliability of the target detection result.

[0004] To achieve the above purpose, one aspect of the present application proposes a target detection method based on multi-sensor fusion, which comprises:

[0005] Obtaining radar point cloud data and multi-view image data of the vehicle surrounding environment, the multi-view image data being collected by a plurality of camera devices arranged in different directions on the vehicle;

[0006] Performing perspective division and feature extraction on the radar point cloud data to obtain multi-view radar point cloud features and multi-view radar point cloud semantic features;

[0007] Combining the multi-view radar point cloud features and a plurality of camera parameter matrices corresponding to the plurality of camera devices, performing feature extraction and depth prediction on the multi-view image data to obtain multi-view image semantic features and multi-view image depth information;

[0008] According to the multi-view image depth information, fusing the multi-view image semantic features and the multi-view radar point cloud semantic features to obtain multi-modal fusion features in the bird's eye view perspective.

[0009] perform target detection on the multi-modal fusion feature to obtain a target detection result.

[0010] Further, the perspective division and feature extraction on the radar point cloud data to obtain multi-perspective radar point cloud features and multi-perspective radar point cloud semantic features include:

[0011] performing division on the radar point cloud data according to a plurality of field of view ranges corresponding to the plurality of camera devices to obtain multi-perspective radar point cloud data;

[0012] performing voxelization feature extraction and fusion on the multi-perspective radar point cloud data to obtain multi-perspective radar columnar point cloud features;

[0013] converting the multi-perspective radar columnar point cloud features to an aerial view perspective to perform multi-scale feature extraction and fusion to obtain the multi-perspective radar point cloud features;

[0014] performing semantic feature extraction on the multi-perspective radar point cloud features to obtain the multi-perspective radar point cloud semantic features.

[0015] Further, the feature extraction and depth prediction on the multi-perspective image data in combination with the multi-perspective radar point cloud features and a plurality of camera parameter matrices corresponding to the plurality of camera devices to obtain multi-perspective image semantic features and multi-perspective image depth information include:

[0016] performing multi-scale feature extraction and fusion on the multi-perspective image data to obtain multi-perspective image features;

[0017] performing gradient extraction on the multi-perspective image features to obtain multi-perspective gradient information;

[0018] performing edge enhancement on the multi-perspective image features according to the multi-perspective gradient information and the multi-perspective radar point cloud features to obtain multi-perspective image enhanced features;

[0019] performing semantic feature extraction and depth prediction on the multi-perspective image enhanced features in combination with a plurality of camera parameter matrices corresponding to the plurality of camera devices to obtain the multi-perspective image semantic features and the multi-perspective image depth information.

[0020] Further, the multi-perspective gradient information includes multi-perspective horizontal gradient values and multi-perspective vertical gradient values; and the edge enhancement on the multi-perspective image features according to the multi-perspective gradient information and the multi-perspective radar point cloud features to obtain multi-perspective image enhanced features includes:

[0021] Based on the multi-view horizontal gradient value and the multi-view vertical gradient value, preliminary edge enhancement is performed on the multi-view image features to obtain preliminary enhanced features of the multi-view image.

[0022] The multi-view image enhancement features are obtained by cross-attention fusion of the preliminary enhancement features of the multi-view image and the multi-view radar point cloud features.

[0023] Furthermore, the step of fusing the semantic features of the multi-view images and the semantic features of the multi-view radar point cloud based on the depth information of the multi-view images to obtain multimodal fusion features in the bird's-eye view includes:

[0024] Based on the depth information of the multi-view image, the semantic features of the multi-view image are expanded in dimension;

[0025] The multi-view image semantic features with expanded dimensions and the multi-view radar point cloud semantic features are fused to obtain the corresponding image bird's-eye view features and point cloud bird's-eye view features.

[0026] A multi-feature fusion network is used to extract and fuse the bird's-eye view features of the image and the bird's-eye view features of the point cloud at multiple scales to obtain the multi-modal fusion features.

[0027] Furthermore, the multi-feature fusion network includes a multi-feature cascade module and a feature fusion module. The feature fusion module includes multiple stacked improved transformers, each of which is obtained by replacing the multi-head self-attention layer in the base transformer with a multi-scale deformable large kernel attention layer. The multi-feature fusion network is used to extract and fuse the image bird's-eye view features and the point cloud bird's-eye view features at multiple scales to obtain the multi-modal fused features, including:

[0028] The multi-feature concatenation module concatenates the image bird's-eye view features and the point cloud bird's-eye view features and performs channel mapping dimensionality reduction to obtain multi-modal concatenated features;

[0029] The feature fusion module performs multi-scale feature extraction and fusion on the multimodal cascaded features to obtain the multimodal fused features.

[0030] Furthermore, the multi-scale deformable large kernel attention layer includes a first convolutional layer, a second convolutional layer, and multiple deformable large kernel attention layers corresponding to multiple scales;

[0031] When the multi-scale deformable large kernel attention layer receives input features, it expands the dimensions of the input features through the first convolutional layer to obtain multiple initial features corresponding to the multiple scales.

[0032] By extracting features from the initial features corresponding to the same scale through each of the deformable large kernel attention layers, intermediate features corresponding to the same scale are obtained.

[0033] The second convolutional layer fuses the intermediate features corresponding to the multiple scales output by the multiple deformable large kernel attention layers to obtain the output features.

[0034] To achieve the above objectives, another aspect of this application proposes a target detection system based on multi-sensor fusion, the system comprising:

[0035] The acquisition module is used to acquire radar point cloud data and multi-view image data of the environment around the vehicle. The multi-view image data is acquired by multiple camera devices set on the vehicle facing different directions.

[0036] The first processing module is used to perform perspective segmentation and feature extraction on the radar point cloud data to obtain multi-view radar point cloud features and multi-view radar point cloud semantic features.

[0037] The second processing module is used to combine the multi-view radar point cloud features and the multiple camera parameter matrices corresponding to the multiple camera devices to perform feature extraction and depth prediction on the multi-view image data, so as to obtain multi-view image semantic features and multi-view image depth information.

[0038] The third processing module is used to fuse the semantic features of the multi-view image and the semantic features of the multi-view radar point cloud based on the depth information of the multi-view image to obtain multimodal fusion features under the bird's-eye view.

[0039] The detection module is used to perform target detection on the multimodal fusion features and obtain the target detection result.

[0040] To achieve the above objectives, another aspect of this application proposes an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0041] To achieve the above objectives, another aspect of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0042] This application includes at least the following beneficial effects: By relying on radar point cloud data with reliable depth information and related multiple camera parameter matrices, reliable depth information can be predicted for multi-view image data, thereby improving the accuracy of multi-view image features when transformed to a bird's-eye view. This is beneficial for improving the reliability of subsequent target detection using multi-modal fusion features formed under the bird's-eye view. By introducing a multi-scale deformable large kernel attention mechanism into the multi-feature fusion network, it is possible to more fully align image bird's-eye features and point cloud bird's-eye features with significant differences at multiple scales, thereby achieving better multi-modal feature fusion results. Attached Figure Description

[0043] Figure 1 This is a flowchart of a target detection method based on multi-sensor fusion provided in an embodiment of this application;

[0044] Figure 2 This is a schematic diagram of the first multi-feature fusion network provided in the embodiments of this application;

[0045] Figure 3 This is a schematic diagram of the structure of a target detection system based on multi-sensor fusion provided in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of systems and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0048] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0049] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0051] Achieving reliable 3D perception is crucial for autonomous driving and a prerequisite for vehicles to operate normally in complex and ever-changing environments. In autonomous driving perception scenarios, multiple sensors are primarily used to acquire information about the vehicle's surroundings. Deep learning technology is then combined to fuse and analyze this data, enabling target recognition and localization, thereby completing tasks such as 3D target detection, semantic segmentation, and target tracking. Camera devices and related radar devices (such as millimeter-wave radar and lidar) are two types of low-cost, heterogeneous sensors often combined in autonomous driving perception scenarios to adapt well to various complex environments.

[0052] In the problem of multi-sensor data fusion, the point cloud data collected by radar equipment contains three-dimensional spatial location information, making it easy to directly convert the point cloud data into a bird's-eye view (BEV) representation. However, the two-dimensional images collected by camera equipment lack depth information. The Line Segment Slope (LSS) algorithm is typically used to predict the depth information corresponding to each pixel in the two-dimensional image to convert it into a bird's-eye view representation. However, the lack of reliable depth information support when applying the LSS algorithm leads to an overall bias in image depth estimation, making the image features less accurate when converted to a bird's-eye view, ultimately affecting the reliability of target detection results.

[0053] Furthermore, due to the significant differences between image features and radar point cloud features, image features mainly contain almost complete geometric structure information, texture and color details of objects, while radar point cloud features mainly contain the edges and surface features of objects (such as the edges of vehicles). If multimodal feature interaction fusion is performed at a single scale, it is often not robust enough when dealing with problems such as extreme shape or scale changes.

[0054] In view of this, embodiments of this application provide a target detection method, system, device, and medium based on multi-sensor fusion. This scheme, relying on radar point cloud data with reliable depth information and related multiple camera parameter matrices, can predict reliable depth information for multi-view image data, thereby improving the accuracy of multi-view image features when converted to a bird's-eye view. This is beneficial for improving the reliability of subsequent target detection using multi-modal fusion features formed in the bird's-eye view. By introducing a multi-scale deformable large kernel attention mechanism into the multi-feature fusion network, it is possible to more fully align image bird's-eye features and point cloud bird's-eye features with significant differences at multiple scales, thereby achieving better multi-modal feature fusion results.

[0055] The target detection method based on multi-sensor fusion provided in this application relates to the field of computer vision technology and can be applied to terminals, servers, or software running on terminals or servers. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the above-mentioned target detection method based on multi-sensor fusion, but is not limited to the above forms.

[0056] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0057] Figure 1 This is an optional flowchart of a target detection method based on multi-sensor fusion provided in an embodiment of this application.Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0058] Step S101: Acquire radar point cloud data and multi-view image data of the vehicle's surrounding environment. The multi-view image data is obtained by collecting data from multiple camera devices set up on the vehicle facing different directions.

[0059] Step S102: Perform viewpoint segmentation and feature extraction on the radar point cloud data to obtain multi-view radar point cloud features and multi-view radar point cloud semantic features.

[0060] Step S103: Combine the multi-view radar point cloud features and the multiple camera parameter matrices corresponding to multiple camera devices to perform feature extraction and depth prediction on the multi-view image data to obtain multi-view image semantic features and multi-view image depth information.

[0061] Step S104: Based on the depth information of the multi-view images, fuse the semantic features of the multi-view images and the semantic features of the multi-view radar point clouds to obtain the multimodal fusion features under the bird's-eye view.

[0062] Step S105: Perform target detection on the multimodal fusion features to obtain the target detection results.

[0063] In step S101 of some embodiments, multiple camera devices are set at different positions on the vehicle to achieve all-round coverage of the vehicle's surrounding environment. The multiple camera devices include, but are not limited to, front-facing camera devices, side-facing camera devices, and rear-facing camera devices. The front-facing camera device is used to capture road condition information and environmental information in front of the vehicle, the side-facing camera device is used to capture road condition information and environmental information to the side of the vehicle, and the rear-facing camera device is used to capture road condition information and environmental information behind the vehicle.

[0064] In step S101 of some embodiments, the radar point cloud data is collected by a millimeter-wave radar device or a lidar device installed on the top of the vehicle, and the relevant radar device is preferably installed at the center of the top of the vehicle to better scan the surrounding environment of the vehicle.

[0065] Steps S101 to S105 as shown in the embodiments of this application rely on radar point cloud data with reliable depth information and related multiple camera parameter matrices, which can predict reliable depth information for multi-view image data, thereby improving the accuracy of multi-view image features when converted to a bird's-eye view, and helping to improve the reliability of subsequent target detection of multi-modal fusion features formed in the bird's-eye view.

[0066] In some embodiments, step S102 may include, but is not limited to, steps S201 to S204.

[0067] Step S201: Divide the radar point cloud data according to the multiple fields of view corresponding to multiple camera devices to obtain multi-view radar point cloud data.

[0068] Step S202: Voxelized feature extraction and fusion are performed on the multi-view radar point cloud data to obtain multi-view radar columnar point cloud features.

[0069] Step S203: Convert the multi-view radar columnar point cloud features to a bird's-eye view for multi-scale feature extraction and fusion to obtain multi-view radar point cloud features.

[0070] Step S204: Extract semantic features from the multi-view radar point cloud features to obtain the semantic features of the multi-view radar point cloud.

[0071] In step S201 of some embodiments, for any camera device, all point cloud data whose position information falls within the field of view corresponding to the camera device are filtered from the radar point cloud data, and the filtered point cloud data are used to form single-view radar point cloud data associated with the camera device; according to this implementation, multi-view radar point cloud data associated with multiple camera devices can be formed.

[0072] In step S202 of some embodiments, for single-view radar point cloud data associated with any camera device, firstly, the field of view corresponding to the camera device is divided into voxels according to a preset voxel grid size to obtain several voxel grids. Secondly, all point cloud data contained in the single-view radar point cloud data are mapped to the divided voxel grids according to their respective position information. Then, for all voxel grids containing at least one point cloud data, relevant feature information is assigned to each voxel grid. Then, all relevant feature information assigned to all voxel grids is concatenated to obtain a longer feature vector. Finally, the feature vector is processed by a fully connected layer and a pooling layer in sequence to obtain the single-view radar columnar point cloud feature associated with the camera device. According to this implementation method, multi-view radar columnar point cloud features associated with multiple camera devices can be formed.

[0073] The relevant feature information assigned to each voxel grid includes voxel center features, point cluster center features, and point cloud features, as detailed below:

[0074] Set the size of each voxel grid to (Δx, Δy, Δz), where Δx is the length of the voxel grid, Δy is the width of the voxel grid, and Δz is the height of the voxel grid.

[0075] For any voxel grid containing point cloud data, if the position index of this voxel grid in the divided voxel grid is (i,j,k), the voxel center corresponding to this voxel grid is calculated as follows:

[0076]

[0077] The center of the point cluster corresponding to this voxel mesh is calculated as follows:

[0078]

[0079] In the formula, N is the number of all point cloud data contained in the voxel mesh, (x n ,y n ,z n ) represents the location information of the nth point cloud data contained in the voxel mesh;

[0080] The voxel center feature assigned to the voxel grid is the geometric distribution feature of each point cloud data contained in the voxel grid relative to the voxel center. The point cluster center feature assigned to the voxel grid is the geometric distribution feature of each point cloud data contained in the voxel grid relative to the point cluster center. The point cloud feature assigned to the voxel grid is the position information and intensity information carried by each point cloud data contained in the voxel grid.

[0081] In step S203 of some embodiments, for the single-view radar bar graph point cloud features associated with any camera device, the single-view radar bar graph point cloud features are first converted to a bird's-eye view for representation to generate a two-dimensional feature map. Then, a point cloud backbone network is used to extract features from the two-dimensional feature map to obtain multi-scale radar point cloud features. Finally, a point cloud SecondFPN network is used to fuse the multi-scale radar point cloud features to obtain the single-view radar point cloud features associated with the camera device. According to this implementation, multi-view radar point cloud features associated with multiple camera devices can be obtained.

[0082] The point cloud SecondFPN network can be understood as a variant of FPN (Feature Pyramid Network). When performing regular point cloud data processing tasks, the point cloud backbone network is usually used to perform the first stage of multi-scale feature extraction, and the point cloud SecondFPN network is used to perform the second stage of multi-scale feature fusion.

[0083] The process of converting the single-view radar bar graph feature to a bird's-eye view can be understood as retaining only the X-axis and Y-axis coordinates of each feature information contained in the single-view radar bar graph feature. Specifically, a zero-based two-dimensional feature map of size (H, W) is first initialized. Then, each feature information contained in the single-view radar bar graph feature is filled into the zero-based two-dimensional feature map according to its position information (i.e., X-axis and Y-axis coordinates). If multiple feature information appears at a certain position in the zero-based two-dimensional feature map, a single feature information is generated by averaging or directly accumulating these multiple feature information to replace and fill that position. Following this filling method, a two-dimensional feature map of size (H, W, C) is finally obtained, where C is the number of feature channels.

[0084] In step S204 of some embodiments, for the single-view radar point cloud features associated with any camera device, multiple convolutional layers are used to process the single-view radar point cloud features (which can be understood as performing multiple convolution operations on the single-view radar point cloud features) to obtain the semantic features of the single-view radar point cloud associated with the camera device; according to this implementation, the semantic features of multi-view radar point clouds associated with multiple camera devices can be obtained.

[0085] Steps S201 to S204 as shown in the embodiments of this application divide the originally acquired radar point cloud data into multi-view radar point cloud data associated with multiple camera devices by relying on multiple fields of view corresponding to multiple camera devices, and further complete point cloud feature extraction and point cloud semantic feature extraction. This can provide reliable data support for subsequent tasks of depth prediction and multimodal feature data fusion based on multi-view image data.

[0086] In some embodiments, step S103 may include, but is not limited to, steps S301 to S304.

[0087] Step S301: Perform multi-scale feature extraction and fusion on the multi-view image data to obtain multi-view image features;

[0088] Step S302: Extract gradients from the features of the multi-view image to obtain multi-view gradient information;

[0089] Step S303: Based on the multi-view radar point cloud features and the multi-view gradient information, perform edge enhancement on the multi-view image features to obtain multi-view image enhancement features;

[0090] Step S304: Combining the multiple camera parameter matrices corresponding to multiple camera devices, perform semantic feature extraction and depth prediction on the multi-view image enhancement features to obtain multi-view image depth information and multi-view image semantic features.

[0091] In step S301 of some embodiments, for single-view image data acquired by any camera device, an image backbone network is used to extract features from the single-view image data to obtain multi-scale image features, and then an image secondFPN network is used to fuse the multi-scale image features to obtain the single-view image features associated with the camera device; according to this implementation, multi-view image features associated with multiple camera devices can be obtained.

[0092] The image SecondFPN network can be understood as a variant of FPN. When performing conventional image data processing tasks, the image backbone network is usually used to perform the first-stage multi-scale feature extraction task, and the image SecondFPN network is used to perform the second-stage multi-scale feature fusion task. The image backbone network preferably adopts the existing ResNet50 network.

[0093] In step S302 of some embodiments, for any single-view image feature associated with a camera device, the Sobel operator is used to extract the gradient value in the horizontal direction from the single-view image feature and record it as the single-view horizontal gradient value, and the Sobel operator is used to extract the gradient value in the vertical direction from the single-view image feature and record it as the single-view vertical gradient value, so as to identify the region in the single-view image data acquired by the camera device that has undergone significant changes, such as object boundaries; according to this implementation, multi-view gradient information associated with multiple camera devices can be obtained, and the multi-view gradient information includes multi-view horizontal gradient values ​​and multi-view vertical gradient values.

[0094] In step S304 of some embodiments, for any single-view image enhancement feature associated with a camera device, a BN (Batch Normalization) layer is used to process the camera parameter matrix corresponding to the camera device to obtain a first camera parameter matrix. A depth feature extraction network is used to process the first camera parameter matrix and the single-view image enhancement feature to obtain the single-view image depth information associated with the camera device. A semantic feature extraction network is used to process the first camera parameter matrix and the single-view image enhancement feature to obtain the single-view image semantic features associated with the camera device. According to this implementation, multi-view image depth information and multi-view image semantic features associated with multiple camera devices can be obtained.

[0095] It should be noted that the deep feature extraction network and the semantic feature extraction network have the same network architecture. The difference lies in the training method used in the early stages, which enables these two networks to extract different features.

[0096] The deep feature extraction network includes a first KAN (Kernelized Attention Network) layer and a first SELayer (Squeeze-and-Excitation Layer). The first KAN layer processes the first camera parameter matrix to obtain the first camera parameter vector. The first SELayer processes the first camera parameter vector and the single-view image enhancement features to obtain initial image depth information. By enhancing feature channels related to depth distribution and suppressing other feature channels in the first SELayer, the network is indirectly helped to extract depth information better. Finally, convolution operation is performed on the initial image depth information to obtain the single-view image depth information.

[0097] The semantic feature extraction network includes a second KAN layer and a second SELayer. The second KAN layer is used to process the first camera parameter matrix to obtain the second camera parameter vector. The second SELayer is used to process the second camera parameter vector and the single-view image enhancement features to obtain the initial image semantic features. By enhancing the feature channels related to semantic information and suppressing other feature channels in the second SELayer, the network can be indirectly helped to extract semantic features better. Finally, the initial image semantic features are convolved to obtain the single-view image semantic features.

[0098] It should be noted that the camera parameter matrix for each camera device typically refers to the camera intrinsic parameter matrix, which includes the camera's focal length, principal point position, image distortion information, etc. This image distortion information includes image rotation and translation information. The camera parameter matrix is ​​beneficial for image depth prediction, providing geometric information to convert image coordinates into camera coordinates, which is helpful for recovering three-dimensional depth from two-dimensional images. Furthermore, by converting between image scale and real-world scale, it can correct disparity maps to calculate image depth, thereby eliminating the effects of image distortion.

[0099] Steps S301 to S304, as shown in the embodiments of this application, enhance the edges of image features by utilizing the relevant gradient information contained in radar point cloud features and image features, thereby improving the grasp of the boundary and position of target objects in the image. Subsequently, the semantic features and depth information of the image are extracted from the edge-enhanced image features by combining the camera parameter matrix, which can provide reliable data support for subsequent multimodal feature data fusion tasks.

[0100] In some embodiments, step S303 may include, but is not limited to, steps S401 to S402.

[0101] Step S401: Based on the multi-view horizontal gradient value and the multi-view vertical gradient value, perform preliminary edge enhancement on the multi-view image features to obtain preliminary enhanced features of the multi-view image.

[0102] Step S402: Perform cross-attention fusion of the preliminary enhancement features of the multi-view image and the multi-view radar point cloud features to obtain the multi-view image enhancement features.

[0103] In step S401 of some embodiments, for any single-view image feature associated with a camera device, based on the single-view horizontal gradient value and single-view vertical gradient value associated with the camera device, the edge features in the single-view image data acquired by the camera device are enhanced and highlighted using the following expression to determine the first image enhancement feature:

[0104]

[0105] In the formula, CI represents the first image enhancement feature, and G... x G represents the horizontal gradient value for this single viewpoint. y CI is the vertical gradient value for this single viewpoint. o This is a feature of the single-view image;

[0106] The first image enhancement feature is processed by a ReLU (Rectified Linear Unit) function and a BN layer to obtain a second image enhancement feature. The second image enhancement feature is then convolved once and added to the single-view image feature to obtain a third image enhancement feature. The third image enhancement feature is then convolved once and processed by a ReLU function and a BN layer to obtain the preliminary single-view image enhancement feature associated with the camera device. According to this embodiment, preliminary multi-view image enhancement features associated with multiple camera devices can be obtained.

[0107] In step S402 of some embodiments, for any single-view image preliminary enhancement features and single-view radar point cloud features associated with a camera device, a cross-attention mechanism is used to fuse the single-view image preliminary enhancement features and the single-view radar point cloud features, so that the two different features establish a correspondence in space, and the radar point cloud features are used to further enhance the image edge feature information, thereby obtaining the single-view image enhancement features associated with the camera device; according to this implementation, multi-view image enhancement features associated with multiple camera devices can be obtained.

[0108] In some embodiments, step S104 may include, but is not limited to, steps S501 to S503.

[0109] Step S501: Based on the depth information of the multi-view image, expand the dimension of the semantic features of the multi-view image;

[0110] Step S502: Perform view feature fusion on the semantic features of the multi-view radar point cloud and the semantic features of the multi-view image after dimensional expansion to obtain the corresponding point cloud bird's-eye view features and image bird's-eye view features.

[0111] Step S503: Extract and fuse multi-scale features of the point cloud bird's-eye view features and the image bird's-eye view features through a multi-feature fusion network to obtain multi-modal fusion features.

[0112] In step S501 of some embodiments, for the single-view image semantic features and single-view image depth information associated with any camera device, the single-view image semantic features and single-view image depth information are multiplied to complete the dimensional expansion, that is, to elevate the two-dimensional image to three-dimensional space; according to this implementation, the dimensional expansion of the multi-view image semantic features associated with multiple camera devices can be performed.

[0113] In step S502 of some embodiments, since the multi-view radar point cloud semantic features have already been represented in a bird's-eye view, the multi-view image semantic features with expanded dimensions are then represented in a bird's-eye view to generate the final multi-view image semantic features. Since the multi-view radar point cloud semantic features and the final multi-view image semantic features have the same size, they are channel-concatenated to obtain multi-view concatenated features associated with multiple camera devices. Because the viewpoints of the camera devices are similar to trapezoids, there may be some overlap in the concatenated features under different viewpoints. Therefore, a larger bird's-eye view feature map is initialized to allow coverage of the multi-view concatenated features. Several grid spaces are divided on the bird's-eye view feature map. Each feature information contained in the multi-view cascaded feature is filled into these grid spaces according to its own position information (i.e., X-axis coordinate value and Y-axis coordinate value). When multiple feature information overlaps in a certain grid space, these multiple feature information are averaged and pooled to obtain a single feature information to replace and fill the grid space. The final bird's-eye view feature map integrating all feature information is obtained according to this filling method. Since the radar point cloud semantic features and image semantic features are initially only channel cascaded and do not involve channel interaction operations, each feature information contained in the final bird's-eye view feature map is directly separated, thus forming point cloud bird's-eye view features and image bird's-eye view features.

[0114] Specifically, for any single-view image semantic feature associated with a camera device, the semantic feature is transformed into a bird's-eye view for representation. This can be understood as retaining only the X-axis and Y-axis coordinates corresponding to each feature information contained in the single-view image semantic feature. In particular, a zero-dimensional two-dimensional feature map of size (H, W) is first initialized. Then, each feature information contained in the single-view image semantic feature is filled into the zero-dimensional two-dimensional feature map according to its position information (i.e., X-axis and Y-axis coordinates). If multiple feature information appears at a certain position in the zero-dimensional two-dimensional feature map, a single feature information is generated by averaging or directly accumulating these multiple feature information to replace and fill that position. Following this filling method, a two-dimensional feature map of size (H, W, C) is finally obtained.

[0115] Specifically, for any single-view radar point cloud semantic features and final single-view image semantic features associated with a camera device, the single-view radar point cloud semantic features and the final single-view image semantic features are concatenated in the channel dimension to obtain the single-view concatenated features associated with the camera device; according to this implementation method, multi-view concatenated features associated with multiple camera devices can be obtained.

[0116] In some embodiments, the multi-feature fusion network mentioned in step S503 above includes a multi-feature cascade module and a feature fusion module. The feature fusion module includes multiple improved transformers stacked together, wherein each improved transformer is obtained by replacing the multi-head self-attention layer set in the base transformer (i.e., the Transformer network) with a multi-scale deformable large kernel attention layer. In specific applications, the multi-feature cascade module is used to cascade and channel map the point cloud bird's-eye view features and the image bird's-eye view features to obtain multi-modal cascaded features, and the feature fusion module is used to extract and fuse multi-scale features from the multi-modal cascaded features to obtain multi-modal fused features.

[0117] The multi-feature cascade module preferably adopts a FACMA (Frequency-Aware Cross-Modal Attention) network. It uses a frequency channel attention mechanism to select all important feature channels from the bird's-eye view features of the point cloud and the bird's-eye view features of the image. Then, it uses a context attention mechanism to adjust the spatial distribution of each feature information falling on all important feature channels, so as to appropriately enhance the weight of each feature information.

[0118] Each improved transformer includes a first normalization layer, a multi-scale deformable large-kernel attention layer, a second normalization layer, and a multilayer perceptron. When the improved transformer receives a first input feature, it processes the first input feature using the first normalization layer to obtain a first feature, processes the first feature using the multi-scale deformable large-kernel attention layer to obtain a second feature, adds the second feature to the first input feature to obtain a third feature, processes the third feature using the second normalization layer to obtain a fourth feature, processes the fourth feature using the multilayer perceptron to obtain a fifth feature, and adds the fifth feature to the third feature to obtain a first output feature. By introducing a multi-scale deformable large-kernel attention mechanism into the improved transformer, it can more effectively focus on specific feature regions, which helps to improve the robustness of the model and enhance the expression of key features.

[0119] The multi-scale deformable large kernel attention layer includes a first convolutional layer, a second convolutional layer, and multiple deformable large kernel attention layers corresponding to multiple scales. When the multi-scale deformable large kernel attention layer receives a second input feature, the first convolutional layer expands the dimension of the second input feature to obtain multiple initial features corresponding to the multiple scales. Each deformable large kernel attention layer extracts features from the initial features corresponding to the same scale to obtain intermediate features corresponding to the same scale. The second convolutional layer fuses the multiple intermediate features corresponding to the multiple scales output by the multiple deformable large kernel attention layers to obtain the second output feature. By implementing multi-scale feature extraction in the multi-scale deformable large kernel attention layer, it is possible to capture more comprehensive features with different modal differences, and it can also adjust the shape to adapt to irregular features.

[0120] Each deformable large kernel attention layer includes a deformable depthwise convolutional layer, a deformable depthwise dilated convolutional layer, a first pointwise convolutional layer, and a second pointwise convolutional layer. When the deformable large kernel attention layer receives a third input feature, it processes the third input feature using the deformable depthwise convolutional layer to obtain a first intermediate feature, processes the first intermediate feature using the deformable depthwise dilated convolutional layer to obtain a second intermediate feature, processes the second intermediate feature using the first pointwise convolutional layer to obtain a third intermediate feature, processes the third input feature using the second pointwise convolutional layer to obtain a fourth intermediate feature, and multiplies the third and fourth intermediate features to obtain a third output feature. By combining depthwise convolution and pointwise convolution operations in the deformable large kernel attention layer, the computational complexity and model parameters can be significantly reduced.

[0121] It should be noted that the first convolutional layer, the second convolutional layer, the first pointwise convolutional layer, and the second pointwise convolutional layer all use convolutional kernels with a size of 1×1.

[0122] In a preferred embodiment, the number of all improved transformers included in the feature fusion module is set to 2, and within each improved transformer, the number of all deformable large kernel attention layers included in the multi-scale deformable large kernel attention layer is set to 3, thereby obtaining the following... Figure 2 The first multi-feature fusion network shown performs the following operations when it receives the point cloud bird's-eye view feature and the image bird's-eye view feature: it uses a multi-feature concatenation module to concatenate and perform channel mapping dimensionality reduction on the point cloud bird's-eye view feature and the image bird's-eye view feature to obtain multi-modal concatenated features; it uses a first improved transformer to perform multi-scale feature extraction and fusion on the multi-modal concatenated features to obtain initial multi-modal fusion features; and it uses a second improved transformer to perform multi-scale feature extraction and fusion on the initial multi-modal fusion features to obtain final multi-modal fusion features.

[0123] In this first improved transformer, the multimodal cascaded features are processed by a first normalization layer to obtain a first intermediate cascaded feature. This first intermediate cascaded feature is then input into a multi-scale deformable large kernel attention layer. The first intermediate cascaded feature is expanded to three times the input dimension by a first convolutional layer to generate three sets of second intermediate cascaded features. These three sets of second intermediate cascaded features are processed by three deformable large kernel attention layers to obtain three sets of corresponding third intermediate cascaded features. These three sets of third intermediate cascaded features are processed by a second convolutional layer to obtain a fourth intermediate cascaded feature. The multimodal cascaded feature and the fourth intermediate cascaded feature are added together to obtain a fifth intermediate cascaded feature. The fifth intermediate cascaded feature is processed by a second normalization layer to obtain a sixth intermediate cascaded feature. The sixth intermediate cascaded feature is processed by a multilayer perceptron to obtain a seventh intermediate cascaded feature. The fifth intermediate cascaded feature and the seventh intermediate cascaded feature are added together to obtain the initial multimodal fusion feature.

[0124] In the second improved transformer, the initial multimodal fusion feature is processed by the first normalization layer to obtain the eighth intermediate concatenated feature. This eighth intermediate concatenated feature is then input into a multi-scale deformable large kernel attention layer. The eighth intermediate concatenated feature is expanded to three times the input dimension by the first convolutional layer to generate three sets of ninth intermediate concatenated features. These three sets of ninth intermediate concatenated features are processed by three deformable large kernel attention layers to obtain three sets of corresponding tenth intermediate concatenated features. These three sets of tenth intermediate concatenated features are processed by the second convolutional layer to obtain the eleventh intermediate concatenated feature. The initial multimodal fusion feature and the eleventh intermediate concatenated feature are added together to obtain the twelfth intermediate concatenated feature. The twelfth intermediate concatenated feature is processed by the second normalization layer to obtain the thirteenth intermediate concatenated feature. The thirteenth intermediate concatenated feature is processed by a multilayer perceptron to obtain the fourteenth intermediate concatenated feature. The twelfth intermediate concatenated feature and the fourteenth intermediate concatenated feature are added together to obtain the final multimodal fusion feature.

[0125] In step S105 of some embodiments, all deep spatial features are extracted from the multimodal fusion features by the backbone network, and all deep spatial features are fused and upsampled by the neck network to obtain a multi-scale feature map. The multi-scale feature map is then used for target detection by the Centerhead detection head, which includes a classification head and a regression head. The classification head is used to predict the heatmap to mark the center position of the target object, and the regression head is used to predict relevant information of the bounding box, including position, size, height, rotation angle, and velocity. Preferably, the backbone network adopts a Residual Network (ResNet) or other similar convolutional neural network architecture, and the neck network adopts a Feature Pyramid Network (FPN).

[0126] Please see Figure 3 This application also provides a target detection system based on multi-sensor fusion, which can implement the above-mentioned target detection method based on multi-sensor fusion. The system includes:

[0127] The acquisition module 601 is used to acquire multi-view image data and radar point cloud data of the environment surrounding the vehicle.

[0128] The multi-view image data was acquired by multiple cameras positioned on the vehicle facing different directions.

[0129] The first processing module 602 is used to perform viewpoint segmentation and feature extraction on the radar point cloud data to obtain multi-view radar point cloud semantic features and multi-view radar point cloud features.

[0130] The second processing module 603 is used to combine the multiple camera parameter matrices corresponding to multiple camera devices and the multi-view radar point cloud features to perform feature extraction and depth prediction on the multi-view image data, so as to obtain multi-view image depth information and multi-view image semantic features.

[0131] The third processing module 604 is used to fuse the semantic features of the multi-view radar point cloud and the semantic features of the multi-view image based on the depth information of the multi-view image, so as to obtain the multi-modal fusion features under the bird's-eye view.

[0132] The detection module 605 is used to perform target detection on multimodal fusion features to obtain target detection results.

[0133] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0134] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned target detection method based on multi-sensor fusion. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0135] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0136] Please see Figure 4 , Figure 4 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0137] The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0138] The memory 702 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this application are implemented through software or firmware, the relevant program code is stored in the memory 702 and is called and executed by the processor 701.

[0139] The input / output interface 703 is used to implement information input and output;

[0140] The communication interface 704 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0141] Bus 705 transmits information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704);

[0142] The processor 701, memory 702, input / output interface 703, and communication interface 704 are connected to each other within the device via bus 705.

[0143] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described target detection method based on multi-sensor fusion.

[0144] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0145] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0146] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0147] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0148] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0150] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0151] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0152] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0153] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0155] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A target detection method based on multi-sensor fusion, characterized in that, The method includes: The radar point cloud data and multi-view image data of the vehicle’s surrounding environment are acquired. The multi-view image data is collected by multiple camera devices set on the vehicle facing different directions. The radar point cloud data is divided into perspectives and features are extracted to obtain multi-view radar point cloud features and multi-view radar point cloud semantic features. By combining the multi-view radar point cloud features and the multiple camera parameter matrices corresponding to the multiple camera devices, feature extraction and depth prediction are performed on the multi-view image data to obtain multi-view image semantic features and multi-view image depth information. Based on the depth information of the multi-view images, the semantic features of the multi-view images and the semantic features of the multi-view radar point clouds are fused to obtain multimodal fusion features under the bird's-eye view. This includes: expanding the dimensions of the semantic features of the multi-view images based on the depth information of the multi-view images; fusing the expanded semantic features of the multi-view images and the semantic features of the multi-view radar point clouds with viewpoint features to obtain corresponding image bird's-eye view features and point cloud bird's-eye view features; and using a multi-feature fusion network to extract and fuse the image bird's-eye view features and point cloud bird's-eye view features at multiple scales to obtain the multimodal fusion features. The modality fusion feature, wherein the multi-feature fusion network includes a multi-feature cascade module and a feature fusion module, the feature fusion module including multiple stacked improved transformers, each of which is obtained by replacing the multi-head self-attention layer set in the base transformer with a multi-scale deformable large kernel attention layer, the multi-feature cascade module cascades and performs channel mapping dimensionality reduction on the image bird's-eye view features and the point cloud bird's-eye view features to obtain multi-modal cascade features, the feature fusion module performs multi-scale feature extraction and fusion on the multi-modal cascade features to obtain the multi-modal fusion feature; The multimodal fusion features are used for target detection to obtain target detection results; The multi-scale deformable large kernel attention layer includes a first convolutional layer, a second convolutional layer, and multiple deformable large kernel attention layers corresponding to multiple scales. When the multi-scale deformable large kernel attention layer receives input features, it expands the dimensions of the input features through the first convolutional layer to obtain multiple initial features corresponding to the multiple scales. Each deformable large kernel attention layer extracts features from the initial features corresponding to the same scale to obtain intermediate features corresponding to the same scale. The second convolutional layer fuses the multiple intermediate features corresponding to the multiple scales output by the multiple deformable large kernel attention layers to obtain output features.

2. The target detection method based on multi-sensor fusion according to claim 1, characterized in that, The step of performing viewpoint segmentation and feature extraction on the radar point cloud data to obtain multi-view radar point cloud features and multi-view radar point cloud semantic features includes: Based on the multiple fields of view corresponding to the multiple camera devices, the radar point cloud data is divided to obtain multi-view radar point cloud data. Voxelized feature extraction and fusion are performed on the multi-view radar point cloud data to obtain multi-view radar columnar point cloud features. The multi-view radar columnar point cloud features are transformed into a bird's-eye view for multi-scale feature extraction and fusion to obtain the multi-view radar point cloud features. Semantic features are extracted from the multi-view radar point cloud features to obtain the semantic features of the multi-view radar point cloud.

3. The target detection method based on multi-sensor fusion according to claim 1, characterized in that, The step of combining the multi-view radar point cloud features and the multiple camera parameter matrices corresponding to the multiple camera devices to perform feature extraction and depth prediction on the multi-view image data to obtain multi-view image semantic features and multi-view image depth information includes: Multi-scale feature extraction and fusion are performed on the multi-view image data to obtain multi-view image features; Gradient extraction is performed on the multi-view image features to obtain multi-view gradient information; Based on the multi-view gradient information and the multi-view radar point cloud features, edge enhancement is performed on the multi-view image features to obtain multi-view image enhancement features. By combining the multiple camera parameter matrices corresponding to the multiple camera devices, semantic feature extraction and depth prediction are performed on the multi-view image enhancement features to obtain the multi-view image semantic features and the multi-view image depth information.

4. The target detection method based on multi-sensor fusion according to claim 3, characterized in that, The multi-view gradient information includes multi-view horizontal gradient values ​​and multi-view vertical gradient values; the step of performing edge enhancement on the multi-view image features based on the multi-view gradient information and the multi-view radar point cloud features to obtain multi-view image enhancement features includes: Based on the multi-view horizontal gradient value and the multi-view vertical gradient value, preliminary edge enhancement is performed on the multi-view image features to obtain preliminary enhanced features of the multi-view image. The multi-view image enhancement features are obtained by cross-attention fusion of the preliminary enhancement features of the multi-view image and the multi-view radar point cloud features.

5. A target detection system based on multi-sensor fusion, characterized in that, The system includes: The acquisition module is used to acquire radar point cloud data and multi-view image data of the environment around the vehicle. The multi-view image data is acquired by multiple camera devices set on the vehicle facing different directions. The first processing module is used to perform perspective segmentation and feature extraction on the radar point cloud data to obtain multi-view radar point cloud features and multi-view radar point cloud semantic features. The second processing module is used to combine the multi-view radar point cloud features and the multiple camera parameter matrices corresponding to the multiple camera devices to perform feature extraction and depth prediction on the multi-view image data, so as to obtain multi-view image semantic features and multi-view image depth information. The third processing module is used to fuse the semantic features of the multi-view images and the semantic features of the multi-view radar point clouds based on the depth information of the multi-view images to obtain multimodal fusion features from a bird's-eye view. This includes: expanding the dimensions of the semantic features of the multi-view images based on the depth information; fusing the expanded semantic features of the multi-view images and the semantic features of the multi-view radar point clouds with viewpoint features to obtain corresponding image bird's-eye view features and point cloud bird's-eye view features; and using a multi-feature fusion network to extract and fuse the image bird's-eye view features and the point cloud bird's-eye view features at multiple scales. The multimodal fusion feature is obtained by the multi-feature fusion network comprising a multi-feature cascade module and a feature fusion module. The feature fusion module comprises multiple stacked improved transformers, each of which is obtained by replacing the multi-head self-attention layer in the base transformer with a multi-scale deformable large kernel attention layer. The multi-feature cascade module performs cascading and channel mapping dimensionality reduction on the image bird's-eye view features and the point cloud bird's-eye view features to obtain multimodal cascaded features. The feature fusion module performs multi-scale feature extraction and fusion on the multimodal cascaded features to obtain the multimodal fusion feature. The detection module is used to perform target detection on the multimodal fusion features to obtain target detection results; The multi-scale deformable large kernel attention layer includes a first convolutional layer, a second convolutional layer, and multiple deformable large kernel attention layers corresponding to multiple scales. When the multi-scale deformable large kernel attention layer receives input features, it expands the dimensions of the input features through the first convolutional layer to obtain multiple initial features corresponding to the multiple scales. Each deformable large kernel attention layer extracts features from the initial features corresponding to the same scale to obtain intermediate features corresponding to the same scale. The second convolutional layer fuses the multiple intermediate features corresponding to the multiple scales output by the multiple deformable large kernel attention layers to obtain output features.

6. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

Images