Target detection method, system and equipment based on multi-sensor fusion and medium

Abstract

The invention provides a target detection method, system and device based on multi-sensor fusion and a medium, and belongs to the technical field of computer vision. The method comprises the following steps: acquiring radar point cloud data and multi-view image data of a surrounding environment of a vehicle; performing view division and feature extraction on the radar point cloud data to obtain multi-view radar point cloud features and multi-view radar point cloud semantic features; performing feature extraction and depth prediction on the multi-view image data in combination with the multi-view radar point cloud features and the plurality of camera parameter matrixes to obtain multi-view image semantic features and multi-view image depth information; according to the multi-view image depth information, fusing the multi-view image semantic features and the multi-view radar point cloud semantic features to obtain multi-modal fusion features under the view angle of the bird's-eye view; and performing target detection on the multi-modal fusion feature to obtain a target detection result. According to the method, the multi-view image features can be more accurately converted to the aerial view view, and the reliability of the target detection result is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a target detection method, system, device and medium based on multi-sensor fusion. Background Art

[0002] In the autonomous driving perception scenario, it mainly relies on a variety of sensors such as camera equipment, millimeter wave radar equipment and lidar equipment to obtain information about the vehicle's surrounding environment, and then combines deep learning technology to fuse various sensor data for analysis to further achieve target recognition and positioning. In the issue of multi-sensor data fusion, the point cloud data collected by the relevant radar equipment itself contains three-dimensional spatial position information, and it is easy to directly convert the point cloud data to a bird's-eye view for representation. However, the two-dimensional image collected by the camera device lacks depth information. The LSS algorithm is usually used to predict the depth information corresponding to each pixel contained in the two-dimensional image to convert the two-dimensional image to a bird's-eye view for representation. However, the lack of reliable depth information support when applying the LSS algorithm will cause an overall deviation in the image depth estimation, making the image features inaccurate when converted to a bird's-eye view, ultimately affecting the reliability of the target detection results. Summary of the invention

[0003] The main purpose of this application is to propose a target detection method, system, device and medium based on multi-sensor fusion, which can predict reliable depth information for multi-view image data to improve the accuracy of multi-view image features when converted to a bird's-eye view, which is conducive to improving the reliability of target detection results.

[0004] To achieve the above objectives, one aspect of the present application proposes a target detection method based on multi-sensor fusion, the method comprising:

[0005] Acquire radar point cloud data and multi-view image data of the vehicle's surrounding environment, wherein the multi-view image data is acquired by collecting data from a plurality of camera devices arranged on the vehicle in different directions;

[0006] Performing perspective division and feature extraction on the radar point cloud data to obtain multi-perspective radar point cloud features and multi-perspective radar point cloud semantic features;

[0007] Combining the multi-view radar point cloud features and the multiple camera parameter matrices corresponding to the multiple camera devices, performing feature extraction and depth prediction on the multi-view image data to obtain multi-view image semantic features and multi-view image depth information;

[0008] According to the multi-view image depth information, the multi-view image semantic features and the multi-view radar point cloud semantic features are fused to obtain a multimodal fusion feature under the bird's-eye view perspective;

[0009] Perform target detection on the multimodal fusion features to obtain a target detection result.

[0010] Furthermore, the performing perspective division and feature extraction on the radar point cloud data to obtain multi-perspective radar point cloud features and multi-perspective radar point cloud semantic features includes:

[0011] Dividing the radar point cloud data according to the multiple field of view ranges corresponding to the multiple camera devices to obtain multi-view radar point cloud data;

[0012] Performing voxel feature extraction and fusion on the multi-view radar point cloud data to obtain multi-view radar columnar point cloud features;

[0013] Convert the multi-view radar columnar point cloud features to a bird's-eye view to perform multi-scale feature extraction and fusion to obtain the multi-view radar point cloud features;

[0014] Semantic features are extracted from the multi-view radar point cloud features to obtain semantic features of the multi-view radar point cloud.

[0015] Further, the combining the multi-view radar point cloud features and the multiple camera parameter matrices corresponding to the multiple camera devices to extract features and predict depth of the multi-view image data to obtain multi-view image semantic features and multi-view image depth information includes:

[0016] Performing multi-scale feature extraction and fusion on the multi-view image data to obtain multi-view image features;

[0017] Performing gradient extraction on the multi-view image features to obtain multi-view gradient information;

[0018] According to the multi-view gradient information and the multi-view radar point cloud features, edge enhancement is performed on the multi-view image features to obtain multi-view image enhancement features;

[0019] In combination with a plurality of camera parameter matrices corresponding to the plurality of camera devices, semantic feature extraction and depth prediction are performed on the multi-view image enhancement features to obtain the multi-view image semantic features and the multi-view image depth information.

[0020] Further, the multi-view gradient information includes a multi-view horizontal gradient value and a multi-view vertical gradient value; the edge enhancement of the multi-view image feature is performed according to the multi-view gradient information and the multi-view radar point cloud feature, and the multi-view image enhancement feature is obtained, which includes:

[0021] Performing preliminary edge enhancement on the multi-view image features according to the multi-view horizontal gradient values ​​and the multi-view vertical gradient values ​​to obtain preliminary enhanced multi-view image features;

[0022] The multi-view image preliminary enhancement features and the multi-view radar point cloud features are cross-attentively fused to obtain the multi-view image enhancement features.

[0023] Furthermore, the multi-view image semantic features and the multi-view radar point cloud semantic features are fused according to the multi-view image depth information to obtain a multimodal fusion feature at a bird's-eye view perspective, including:

[0024] According to the multi-view image depth information, dimensionally expanding the multi-view image semantic features;

[0025] Performing perspective feature fusion on the multi-view image semantic features after dimension expansion and the multi-view radar point cloud semantic features to obtain corresponding image bird's-eye view features and point cloud bird's-eye view features;

[0026] A multi-feature fusion network is used to perform multi-scale feature extraction and fusion on the image bird's-eye view features and the point cloud bird's-eye view features to obtain the multimodal fusion features.

[0027] Furthermore, the multi-feature fusion network includes a multi-feature cascade module and a feature fusion module, the feature fusion module includes a plurality of stacked improved transformers, each of which is obtained by replacing the multi-head self-attention layer set in the basic transformer with a multi-scale deformable large core attention layer; the multi-feature fusion network is used to perform multi-scale feature extraction and fusion on the image bird's-eye view feature and the point cloud bird's-eye view feature, and the multimodal fusion feature obtained includes:

[0028] The image bird's-eye view features and the point cloud bird's-eye view features are cascaded and channel-mapped for dimensionality reduction by the multi-feature cascade module to obtain a multimodal cascade feature;

[0029] The multi-scale feature extraction and fusion of the multi-modal cascade features are performed through the feature fusion module to obtain the multi-modal fusion features.

[0030] Furthermore, the multi-scale deformable large core attention layer includes a first convolutional layer, a second convolutional layer, and a plurality of deformable large core attention layers correspondingly arranged at a plurality of scales;

[0031] When the multi-scale deformable large core attention layer receives an input feature, the input feature is dimensionally expanded by the first convolutional layer to obtain a plurality of initial features corresponding to the plurality of scales;

[0032] Extracting the initial features corresponding to the same scale through each deformable large core attention layer, and obtaining the intermediate features corresponding to the same scale;

[0033] The second convolutional layer is used to fuse the multiple intermediate features corresponding to the multiple scales output by the multiple deformable large core attention layers to obtain output features.

[0034] To achieve the above object, another aspect of the present application proposes a target detection system based on multi-sensor fusion, the system comprising:

[0035] An acquisition module, used to acquire radar point cloud data and multi-view image data of the vehicle's surrounding environment, wherein the multi-view image data is acquired by collecting multiple camera devices arranged on the vehicle in different directions;

[0036] A first processing module is used to perform perspective division and feature extraction on the radar point cloud data to obtain multi-perspective radar point cloud features and multi-perspective radar point cloud semantic features;

[0037] A second processing module is used to combine the multi-view radar point cloud features and the multiple camera parameter matrices corresponding to the multiple camera devices to perform feature extraction and depth prediction on the multi-view image data to obtain multi-view image semantic features and multi-view image depth information;

[0038] A third processing module is used to fuse the multi-view image semantic features and the multi-view radar point cloud semantic features according to the multi-view image depth information to obtain a multimodal fusion feature under the bird's-eye view perspective;

[0039] The detection module is used to perform target detection on the multimodal fusion features to obtain a target detection result.

[0040] To achieve the above objective, another aspect of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0041] To achieve the above objective, another aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0042] The present application includes at least the following beneficial effects: by relying on radar point cloud data with reliable depth information and related multiple camera parameter matrices, reliable depth information can be predicted for multi-view image data to improve the accuracy of multi-view image features when converted to a bird's-eye view, which is conducive to improving the reliability of subsequent target detection of multimodal fusion features formed under the bird's-eye view. By introducing a multi-scale deformable large core attention mechanism in the multi-feature fusion network, it is possible to more fully align image bird's-eye view features and point cloud bird's-eye view features with large differences at multiple scales to achieve better multimodal feature fusion effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of a target detection method based on multi-sensor fusion provided in an embodiment of the present application;

[0044] Figure 2 is a schematic diagram of a first multi-feature fusion network provided in an embodiment of the present application;

[0045] Figure 3 is a schematic diagram of the structure of a target detection system based on multi-sensor fusion provided in an embodiment of the present application;

[0046] Figure 4 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0048] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0049] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0051] Achieving reliable 3D perception is the key to autonomous driving and a prerequisite for vehicles to drive normally in complex and changing environments. In autonomous driving perception scenarios, it mainly relies on a variety of sensors to obtain information about the vehicle's surrounding environment, and then combines deep learning technology to fuse various sensor data for analysis to further achieve target recognition and positioning, thereby completing tasks such as 3D target detection, semantic segmentation, and target tracking. Camera equipment and related radar equipment (such as millimeter-wave radar equipment and lidar equipment, etc.) are two types of low-cost and heterogeneous sensors, which are often combined and applied to autonomous driving perception scenarios to adapt well to various complex environments.

[0052] In the problem of multi-sensor data fusion, the point cloud data collected by the relevant radar equipment itself contains three-dimensional spatial position information, and it is easy to directly convert the point cloud data to a bird's eye view (BEV) perspective for representation. However, the two-dimensional images collected by the camera equipment lack depth information. The LSS (Line Segment Slope) algorithm is usually used to predict the depth information corresponding to each pixel contained in the two-dimensional image to convert the two-dimensional image to a bird's eye view for representation. However, the lack of reliable depth information support when applying the LSS algorithm will lead to an overall deviation in the image depth estimation, making the image features inaccurate when converted to a bird's eye view perspective, ultimately affecting the reliability of the target detection results.

[0053] In addition, due to the large difference between image features and radar point cloud features, image features mainly contain almost complete geometric structure information, texture and color details of the object, and radar point cloud features mainly contain the edges and surface features of the object (such as the edges of vehicles, etc.). If multimodal features are interactively fused at a single scale, it is often not robust enough when dealing with problems such as extreme shapes or scale changes.

[0054] In view of this, the embodiments of the present application provide a target detection method, system, device and medium based on multi-sensor fusion. The scheme relies on radar point cloud data with reliable depth information and multiple related camera parameter matrices to predict reliable depth information for multi-view image data, so as to improve the accuracy of multi-view image features when converted to a bird's-eye view, which is conducive to improving the reliability of subsequent target detection of multi-modal fusion features formed under the bird's-eye view. By introducing a multi-scale deformable large core attention mechanism in the multi-feature fusion network, it is possible to more fully align image bird's-eye view features and point cloud bird's-eye view features with large differences at multiple scales to achieve a better multi-modal feature fusion effect.

[0055] The target detection method based on multi-sensor fusion provided in the embodiment of the present application relates to the field of computer vision technology, and can be applied to a terminal, a server, or software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the above-mentioned target detection method based on multi-sensor fusion, etc., but is not limited to the above forms.

[0056] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0057] Figure 1 is an optional flow chart of a target detection method based on multi-sensor fusion provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0058] Step S101, acquiring radar point cloud data and multi-view image data of the vehicle's surrounding environment, where the multi-view image data is acquired by collecting data from a plurality of camera devices arranged on the vehicle in different directions;

[0059] Step S102: performing perspective division and feature extraction on the radar point cloud data to obtain multi-perspective radar point cloud features and multi-perspective radar point cloud semantic features;

[0060] Step S103: combining the multi-view radar point cloud features and the multiple camera parameter matrices corresponding to the multiple camera devices, performing feature extraction and depth prediction on the multi-view image data to obtain the multi-view image semantic features and the multi-view image depth information;

[0061] Step S104: According to the multi-view image depth information, the multi-view image semantic features and the multi-view radar point cloud semantic features are fused to obtain a multimodal fusion feature under the bird's-eye view perspective;

[0062] Step S105: perform target detection on the multimodal fusion features to obtain target detection results.

[0063] In step S101 of some embodiments, multiple camera devices are set at different positions of the vehicle to achieve all-round coverage of the vehicle's surrounding environment. The multiple camera devices include but are not limited to front camera devices, side camera devices and rear camera devices. The front camera device is used to capture road condition information and environmental information in front of the vehicle, the side camera device is used to capture road condition information and environmental information on the side of the vehicle, and the rear camera device is used to capture road condition information and environmental information behind the vehicle.

[0064] In step S101 of some embodiments, radar point cloud data is collected by a millimeter-wave radar device or a lidar device installed on the top of the vehicle, and the relevant radar device is preferably installed at the center of the top of the vehicle to better perform an all-round scan of the vehicle's surrounding environment.

[0065] Steps S101 to S105 shown in the embodiment of the present application rely on radar point cloud data with reliable depth information and multiple related camera parameter matrices, and can predict reliable depth information for multi-view image data to improve the accuracy of multi-view image features when converted to a bird's-eye view, which is beneficial to improving the reliability of subsequent target detection using multi-modal fusion features formed under the bird's-eye view.

[0066] In some embodiments, the above step S102 may include but is not limited to steps S201 to S204.

[0067] Step S201: dividing the radar point cloud data according to multiple field of view ranges corresponding to multiple camera devices to obtain multi-view radar point cloud data;

[0068] Step S202: performing voxel feature extraction and fusion on the multi-view radar point cloud data to obtain multi-view radar columnar point cloud features;

[0069] Step S203: converting the multi-view radar columnar point cloud features to a bird's-eye view to perform multi-scale feature extraction and fusion to obtain multi-view radar point cloud features;

[0070] Step S204: extracting semantic features from the multi-view radar point cloud features to obtain semantic features of the multi-view radar point cloud.

[0071] In step S201 of some embodiments, for any camera device, all point cloud data whose location information falls within the field of view corresponding to the camera device are filtered out from the radar point cloud data, and single-view radar point cloud data associated with the camera device are formed by all the filtered point cloud data; according to this embodiment, multi-view radar point cloud data associated with multiple camera devices can be formed.

[0072] In step S202 of some embodiments, for the single-view radar point cloud data associated with any camera device, the field of view corresponding to the camera device is firstly divided into voxels according to a preset voxel grid size to obtain a plurality of voxel grids, and then all the point cloud data contained in the single-view radar point cloud data are mapped to the divided plurality of voxel grids according to their respective position information, and then for all the voxel grids containing at least one point cloud data, relevant feature information is assigned to each voxel grid, and then all the relevant feature information assigned to all the voxel grids are spliced ​​to obtain a longer feature vector, and finally the feature vector is processed by the fully connected layer and the pooling layer in sequence to obtain the single-view radar columnar point cloud feature associated with the camera device; according to this embodiment, multi-view radar columnar point cloud features associated with multiple camera devices can be formed.

[0073] Among them, the relevant feature information assigned to each voxel grid includes voxel center features, point cluster center features and point cloud features, which are described as follows:

[0074] Set the size of each voxel grid to (Δx, Δy, Δz), where Δx is the length of the voxel grid, Δy is the width of the voxel grid, and Δz is the height of the voxel grid;

[0075] For any voxel grid containing point cloud data, if the position index of the voxel grid in the divided voxel grids is (i, j, k), the voxel center corresponding to the voxel grid is calculated as:

[0076]

[0077] The point cluster center corresponding to the voxel grid is calculated as:

[0078]

[0079] Where N is the number of all point cloud data contained in the voxel grid, (x n ,y n ,z n ) is the position information of the nth point cloud data contained in the voxel grid;

[0080] The voxel center feature assigned by the voxel grid is the geometric distribution feature of each point cloud data contained in the voxel grid relative to the voxel center. The point cluster center feature assigned by the voxel grid is the geometric distribution feature of each point cloud data contained in the voxel grid relative to the point cluster center. The point cloud feature assigned by the voxel grid is the position information of each point cloud data contained in the voxel grid and the intensity information it carries.

[0081] In step S203 of some embodiments, for the single-view radar columnar point cloud features associated with any camera device, the single-view radar columnar point cloud features are first converted to a bird's-eye view to generate a two-dimensional feature map, and then the point cloud Backbone network is used to extract features from the two-dimensional feature map to obtain multi-scale radar point cloud features. Finally, the point cloud SecondFPN network is used to fuse the multi-scale radar point cloud features to obtain the single-view radar point cloud features associated with the camera device. According to this implementation, multi-view radar point cloud features associated with multiple camera devices can be obtained.

[0082] Among them, the point cloud SecondFPN network can be understood as a variant of FPN (Feature Pyramid Network). When performing conventional point cloud data processing tasks, the point cloud Backbone network is usually used to perform the first-stage multi-scale feature extraction task, and the point cloud SecondFPN network is used to perform the second-stage multi-scale feature fusion task.

[0083] Among them, regarding the conversion of the single-view radar cylindrical point cloud feature to the bird's-eye view for representation, it can be understood that only the X-axis coordinate value and the Y-axis coordinate value corresponding to each feature information contained in the single-view radar cylindrical point cloud feature are retained; specifically, first, an all-zero two-dimensional feature map with a size of (H, W) is initialized, and then each feature information contained in the single-view radar cylindrical point cloud feature is filled into the all-zero two-dimensional feature map according to their respective position information (i.e., the X-axis coordinate value and the Y-axis coordinate value). If multiple feature information appears at a certain position in the all-zero two-dimensional feature map, a feature information is generated by averaging or directly accumulating the multiple feature information to replace and fill in the position. According to this filling method, a two-dimensional feature map with a size of (H, W, C) is finally obtained, and C is the number of channels of the feature.

[0084] In step S204 of some embodiments, for the single-view radar point cloud features associated with any camera device, multiple convolution layers are used to process the single-view radar point cloud features (which can be understood as performing multiple convolution operations on the single-view radar point cloud features) to obtain the single-view radar point cloud semantic features associated with the camera device; according to this implementation, multi-view radar point cloud semantic features associated with multiple camera devices can be obtained.

[0085] Steps S201 to S204 shown in the embodiment of the present application, by relying on multiple fields of view corresponding to multiple camera devices, divide the originally acquired radar point cloud data into multi-view radar point cloud data associated with multiple camera devices and further complete point cloud feature extraction and point cloud semantic feature extraction, can provide reliable data support for subsequent depth prediction tasks and multi-modal feature data fusion tasks on multi-view image data.

[0086] In some embodiments, the above step S103 may include but is not limited to steps S301 to S304.

[0087] Step S301, performing multi-scale feature extraction and fusion on the multi-view image data to obtain multi-view image features;

[0088] Step S302: performing gradient extraction on the multi-view image features to obtain multi-view gradient information;

[0089] Step S303: performing edge enhancement on the multi-view image feature according to the multi-view radar point cloud feature and the multi-view gradient information to obtain a multi-view image enhancement feature;

[0090] Step S304: Combine multiple camera parameter matrices corresponding to multiple camera devices to perform semantic feature extraction and depth prediction on the multi-view image enhancement feature to obtain multi-view image depth information and multi-view image semantic features.

[0091] In step S301 of some embodiments, for the single-view image data collected by any camera device, an image Backbone network is used to extract features of the single-view image data to obtain multi-scale image features, and then an image SecondFPN network is used to fuse the multi-scale image features to obtain the single-view image features associated with the camera device; according to this implementation, multi-view image features associated with multiple camera devices can be obtained.

[0092] Among them, the image SecondFPN network can be understood as a variant of FPN. When performing conventional image data processing tasks, the image Backbone network is usually used to perform the first-stage multi-scale feature extraction task, and the image SecondFPN network is used to perform the second-stage multi-scale feature fusion task. The image Backbone network preferably adopts the existing Resnet50 network.

[0093] In step S302 of some embodiments, for a single-view image feature associated with any camera device, a Sobel operator is used to extract a gradient value in the horizontal direction from the single-view image feature and recorded as a single-view horizontal gradient value, and a Sobel operator is used to extract a gradient value in the vertical direction from the single-view image feature and recorded as a single-view vertical gradient value, so as to identify areas where significant changes occur in the single-view image data collected by the camera device, such as object boundaries; according to this embodiment, multi-view gradient information associated with multiple camera devices can be obtained, and the multi-view gradient information includes multi-view horizontal gradient values ​​and multi-view vertical gradient values.

[0094] In step S304 of some embodiments, for the single-view image enhancement features associated with any camera device, a BN (Batch Normalization) layer is used to process the camera parameter matrix corresponding to the camera device to obtain a first camera parameter matrix, a depth feature extraction network is used to process the first camera parameter matrix and the single-view image enhancement features to obtain the single-view image depth information associated with the camera device, and a semantic feature extraction network is used to process the first camera parameter matrix and the single-view image enhancement features to obtain the single-view image semantic features associated with the camera device; according to this implementation, multi-view image depth information and multi-view image semantic features associated with multiple camera devices can be obtained.

[0095] It should be noted that the network architectures of the deep feature extraction network and the semantic feature extraction network are actually the same, but the two networks are equipped with the ability to extract different features through network training in the early stage.

[0096] Among them, the deep feature extraction network includes a first KAN (Kernelized Attention Network) layer and a first SELayer (Squeeze-and-Excitation Layer). The first KAN layer is used to process the first camera parameter matrix to obtain a first camera parameter vector. The first SELayer is used to process the first camera parameter vector and the single-view image enhancement feature to obtain initial image depth information. By enhancing the feature channels related to the depth distribution in the first SELayer and suppressing other feature channels, the network is indirectly helped to better extract the depth information. Finally, a convolution operation is performed on the initial image depth information to obtain the single-view image depth information.

[0097] Among them, the semantic feature extraction network includes a second KAN layer and a second SELayer. The second KAN layer is used to process the first camera parameter matrix to obtain a second camera parameter vector. The second SELayer is used to process the second camera parameter vector and the single-view image enhancement feature to obtain the initial image semantic feature. By enhancing the feature channels related to the semantic information and suppressing other feature channels in the second SELayer, the network is indirectly helped to better extract semantic features. Finally, a convolution operation is performed on the initial image semantic feature to obtain the single-view image semantic feature.

[0098] It should be noted that the camera parameter matrix corresponding to each camera device usually refers to the camera intrinsic parameter matrix, which includes the focal length, principal point position, image distortion information, etc. of the camera device. The image distortion information includes image rotation information and translation information, etc. The camera parameter matrix is ​​beneficial for image depth prediction. It can provide geometric information for converting image coordinates into camera coordinates, which is conducive to recovering three-dimensional depth from two-dimensional images. At the same time, through the conversion between image scale and actual world scale, the disparity map can be corrected to calculate image depth, thereby eliminating the impact of image distortion.

[0099] In steps S301 to S304 shown in the embodiment of the present application, edge enhancement is performed on image features by utilizing relevant gradient information contained in radar point cloud features and image features to improve the grasp of the boundaries and positions of target objects in the image, and then image semantic features and image depth information are correspondingly extracted from the edge-enhanced image features in combination with the camera parameter matrix, thereby providing reliable data support for the subsequent execution of multimodal feature data fusion tasks.

[0100] In some embodiments, the above step S303 may include but is not limited to steps S401 to S402.

[0101] Step S401: performing preliminary edge enhancement on the multi-view image feature according to the multi-view horizontal gradient value and the multi-view vertical gradient value to obtain a preliminary enhanced multi-view image feature;

[0102] Step S402: cross-attentionally fuse the preliminary enhanced features of the multi-view image and the multi-view radar point cloud features to obtain multi-view image enhanced features.

[0103] In step S401 of some embodiments, for a single-view image feature associated with any camera device, the edge feature in the single-view image data collected by the camera device is enhanced and highlighted by the following expression according to the single-view horizontal gradient value and the single-view vertical gradient value associated with the camera device, so as to determine a first image enhancement feature:

[0104]

[0105] Where CI is the first image enhancement feature, G x is the horizontal gradient value of the single viewing angle, G y is the vertical gradient value of the single viewing angle, CI o is the single-view image feature;

[0106] The first image enhancement feature is processed by a ReLU (Rectified Linear Unit) function and a BN layer to obtain a second image enhancement feature, the second image enhancement feature is convolved once and then added to the single-view image feature to obtain a third image enhancement feature, the third image enhancement feature is convolved once and then processed by a ReLU function and a BN layer to obtain a preliminary enhancement feature of the single-view image associated with the camera device; according to this embodiment, preliminary enhancement features of multi-view images associated with multiple camera devices can be obtained.

[0107] In step S402 of some embodiments, for the single-view image preliminary enhancement features and the single-view radar point cloud features associated with any camera device, a cross-attention mechanism is used to fuse the single-view image preliminary enhancement features and the single-view radar point cloud features, so that these two different features establish a corresponding relationship in space, and the radar point cloud features are used to further enhance the image edge feature information, thereby obtaining the single-view image enhancement features associated with the camera device; according to this implementation, multi-view image enhancement features associated with multiple camera devices can be obtained.

[0108] In some embodiments, the above step S104 may include but is not limited to steps S501 to S503.

[0109] Step S501: Dimensionally expand the semantic features of the multi-view image according to the depth information of the multi-view image;

[0110] Step S502: fusing the multi-view radar point cloud semantic features with the multi-view image semantic features after dimension expansion to obtain corresponding point cloud bird's-eye view features and image bird's-eye view features;

[0111] Step S503: extract and fuse the point cloud bird's-eye view features and the image bird's-eye view features at multiple scales through a multi-feature fusion network to obtain multimodal fusion features.

[0112] In step S501 of some embodiments, for the single-view image semantic features and the single-view image depth information associated with any camera device, the single-view image semantic features and the single-view image depth information are multiplied to complete dimensional expansion, that is, the two-dimensional image is lifted to a three-dimensional space; according to this implementation, the multi-view image semantic features associated with multiple camera devices can be dimensionally expanded.

[0113] In step S502 of some embodiments, since the multi-view radar point cloud semantic features have been converted to a bird's-eye view for representation, the multi-view image semantic features after dimension expansion are converted to a bird's-eye view for representation to generate a final multi-view image semantic feature; since the multi-view radar point cloud semantic features and the final multi-view image semantic features have the same size, the multi-view radar point cloud semantic features and the final multi-view image semantic features are channel-concatenated to obtain multi-view cascade features associated with multiple camera devices; since the perspective of the camera device is similar to a trapezoid, there may be partial feature overlap in the cascade features at different perspectives, and a larger bird's-eye view feature map is initialized to allow the multi-view cascade features to be covered. A number of grid spaces are divided on the bird's-eye view feature map, and each feature information contained in the multi-view cascade feature is filled into the several grid spaces according to their respective position information (i.e., X-axis coordinate value and Y-axis coordinate value). When multiple overlapping feature information appear in a certain network space, one feature information is obtained by averaging and pooling the multiple feature information to replace and fill it into the network space, and a final bird's-eye view feature map integrating all feature information is obtained according to this filling method. Since the radar point cloud semantic features and image semantic features are initially only channel cascaded and do not involve channel interaction operations, each feature information contained in the final bird's-eye view feature map is directly separated, thereby forming point cloud bird's-eye view features and image bird's-eye view features.

[0114] Among them, for the dimensionally expanded single-view image semantic features associated with any camera device, the dimensionally expanded single-view image semantic features are converted to a bird's-eye view for representation, which can be understood as only retaining the X-axis coordinate value and the Y-axis coordinate value corresponding to each feature information contained in the dimensionally expanded single-view image semantic features; specifically, firstly, an all-zero two-dimensional feature map of size (H, W) is initialized, and then each feature information contained in the dimensionally expanded single-view image semantic features is filled into the all-zero two-dimensional feature map according to their respective position information (i.e., the X-axis coordinate value and the Y-axis coordinate value). If multiple feature information appear at a certain position in the all-zero two-dimensional feature map, a feature information is generated by averaging or directly accumulating the multiple feature information to replace and fill in the position, and a two-dimensional feature map of size (H, W, C) is finally obtained according to this filling method.

[0115] Among them, for the single-view radar point cloud semantic features and the final single-view image semantic features associated with any camera device, the single-view radar point cloud semantic features and the final single-view image semantic features are cascaded in the channel dimension to obtain the single-view cascade features associated with the camera device; according to this implementation, multi-view cascade features associated with multiple camera devices can be obtained.

[0116] In some embodiments, the multi-feature fusion network mentioned in the above step S503 includes a multi-feature cascade module and a feature fusion module, and the feature fusion module includes multiple improved transformers stacked together, wherein each improved transformer is obtained by replacing the multi-head self-attention layer set in the basic transformer (i.e., the Transformer network) with a multi-scale deformable large core attention layer; in a specific application, the multi-feature cascade module is used to cascade and channel map the point cloud bird's-eye view features and the image bird's-eye view features to obtain multimodal cascade features, and the feature fusion module is used to perform multi-scale feature extraction and fusion on the multimodal cascade features to obtain multimodal fusion features.

[0117] Among them, the multi-feature cascade module preferably adopts a FACMA (Frequency-Aware Cross-ModalAttention) network, and uses a frequency channel attention mechanism to filter out all important feature channels from the point cloud bird's-eye view features and the image bird's-eye view features, and then uses a contextual attention mechanism to adjust the spatial distribution of each feature information falling on all important feature channels to appropriately enhance the weight of each feature information.

[0118] Each improved transformer includes a first normalization layer, a multi-scale deformable large core attention layer, a second normalization layer and a multi-layer perceptron; when the improved transformer receives a first input feature, the first normalization layer is used to process the first input feature to obtain a first feature, the multi-scale deformable large core attention layer is used to process the first feature to obtain a second feature, the second feature is added to the first input feature to obtain a third feature, the second normalization layer is used to process the third feature to obtain a fourth feature, the multi-layer perceptron is used to process the fourth feature to obtain a fifth feature, and the fifth feature is added to the third feature to obtain a first output feature. By introducing a multi-scale deformable large core attention mechanism in the improved transformer, it is possible to more effectively focus on specific feature areas, which helps to improve the robustness of the model and enhance the expression of key features.

[0119] Among them, the multi-scale deformable large core attention layer includes a first convolution layer, a second convolution layer and multiple deformable large core attention layers corresponding to multiple scales; when the multi-scale deformable large core attention layer receives a second input feature, the first convolution layer is used to expand the dimension of the second input feature to obtain multiple initial features corresponding to the multiple scales, each deformable large core attention layer is used to extract the initial features corresponding to the same scale to obtain the intermediate features corresponding to the same scale, and the second convolution layer is used to fuse the multiple intermediate features corresponding to the multiple scales output by the multiple deformable large core attention layers to obtain the second output feature. By realizing multiple scale feature extraction in the multi-scale deformable large core attention layer, the difference features of different modalities can be captured more comprehensively, and the shape can be adjusted to adapt to irregular features.

[0120] Among them, each deformable large core attention layer includes a deformable deep convolution layer, a deformable deep expansion convolution layer, a first point-by-point convolution layer and a second point-by-point convolution layer; when the deformable large core attention layer receives a third input feature, the third input feature is processed by the deformable deep convolution layer to obtain a first intermediate feature, the first intermediate feature is processed by the deformable deep expansion convolution layer to obtain a second intermediate feature, the second intermediate feature is processed by the first point-by-point convolution layer to obtain a third intermediate feature, the third input feature is processed by the second point-by-point convolution layer to obtain a fourth intermediate feature, and the third intermediate feature and the fourth intermediate feature are multiplied to obtain a third output feature. By combining the deep convolution operation and the point-by-point convolution operation in the deformable large core attention layer, the computational complexity and model parameters can be greatly reduced.

[0121] It should be noted that a convolution kernel with a size of 1×1 is used in the first convolution layer, the second convolution layer, the first point-by-point convolution layer, and the second point-by-point convolution layer.

[0122] As a preferred embodiment, the number of all improved transformers included in the feature fusion module is set to 2, and in each improved transformer, the number of all deformable large core attention layers included in the multi-scale deformable large core attention layer is set to 3, thereby obtaining the following: Figure 2 The first multi-feature fusion network shown, when the first multi-feature fusion network receives the point cloud bird's-eye view feature and the image bird's-eye view feature, specifically performs the following operations: uses a multi-feature cascade module to cascade and channel map the point cloud bird's-eye view feature and the image bird's-eye view feature to reduce the dimension, so as to obtain a multimodal cascade feature; uses the first improved transformer to perform multi-scale feature extraction and fusion on the multimodal cascade feature to obtain an initial multimodal fusion feature; uses the second improved transformer to perform multi-scale feature extraction and fusion on the initial multimodal fusion feature to obtain a final multimodal fusion feature.

[0123] In the first improved transformer, the multimodal cascade feature is processed by a first normalization layer to obtain a first intermediate cascade feature, the first intermediate cascade feature is input into a multi-scale deformable large core attention layer, the first convolution layer is used to expand the first intermediate cascade feature to three times the input dimension to generate three groups of second intermediate cascade features, the three groups of second intermediate cascade features are processed by three deformable large core attention layers to obtain three corresponding groups of third intermediate cascade features, the three groups of third intermediate cascade features are processed by a second convolution layer to obtain a fourth intermediate cascade feature, the multimodal cascade feature and the fourth intermediate cascade feature are added to obtain a fifth intermediate cascade feature, the fifth intermediate cascade feature is processed by a second normalization layer to obtain a sixth intermediate cascade feature, the sixth intermediate cascade feature is processed by a multilayer perceptron to obtain a seventh intermediate cascade feature, and the fifth intermediate cascade feature and the seventh intermediate cascade feature are added to obtain the initial multimodal fusion feature.

[0124] In the second improved transformer, the initial multimodal fusion feature is processed by the first normalization layer to obtain the eighth intermediate cascade feature, the eighth intermediate cascade feature is input into the multi-scale deformable large core attention layer, the eighth intermediate cascade feature is expanded to three times the input dimension by the first convolution layer to generate three groups of ninth intermediate cascade features, the three groups of ninth intermediate cascade features are processed by three deformable large core attention layers to obtain the corresponding three groups of tenth intermediate cascade features, the three groups of tenth intermediate cascade features are processed by the second convolution layer to obtain the eleventh intermediate cascade feature, the initial multimodal fusion feature and the eleventh intermediate cascade feature are added to obtain the twelfth intermediate cascade feature, the second normalization layer is used to process the twelfth intermediate cascade feature to obtain the thirteenth intermediate cascade feature, the thirteenth intermediate cascade feature is processed by the multilayer perceptron to obtain the fourteenth intermediate cascade feature, and the twelfth intermediate cascade feature and the fourteenth intermediate cascade feature are added to obtain the final multimodal fusion feature.

[0125] In step S105 of some embodiments, all deep spatial features are extracted from the multimodal fusion features through the backbone network, all deep spatial features are fused and upsampled through the neck network to obtain a multi-scale feature map, and target detection is performed on the multi-scale feature map through the Centerhead detection head, which includes a classification head and a regression head. The classification head is used to predict the heat map to mark the center position of the target object, and the regression head is used to predict relevant information of the bounding box, including position, size, height, rotation angle and speed, etc.; wherein the backbone network preferably adopts a residual network (ResNet) or other similar convolutional neural network architectures, and the neck network preferably adopts a feature pyramid network (FPN).

[0126] See also Figure 3 The embodiment of the present application further provides a target detection system based on multi-sensor fusion, which can implement the above-mentioned target detection method based on multi-sensor fusion, and the system includes:

[0127] An acquisition module 601 is used to acquire multi-view image data and radar point cloud data of the vehicle's surrounding environment;

[0128] The multi-view image data is collected by multiple camera devices arranged on the vehicle in different directions;

[0129] The first processing module 602 is used to perform perspective division and feature extraction on the radar point cloud data to obtain multi-perspective radar point cloud semantic features and multi-perspective radar point cloud features;

[0130] The second processing module 603 is used to combine multiple camera parameter matrices corresponding to multiple camera devices and the multi-view radar point cloud features to perform feature extraction and depth prediction on the multi-view image data to obtain multi-view image depth information and multi-view image semantic features;

[0131] The third processing module 604 is used to fuse the multi-view radar point cloud semantic features and the multi-view image semantic features according to the multi-view image depth information to obtain a multi-modal fusion feature under the bird's-eye view perspective;

[0132] The detection module 605 is used to perform target detection on the multimodal fusion features to obtain target detection results.

[0133] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0134] The embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned target detection method based on multi-sensor fusion when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0135] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0136] See also Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0137] The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0138] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solution provided in the embodiment of the present application is implemented by software or firmware, the relevant program code is stored in the memory 702, and the processor 701 calls and executes the technical solution provided in the embodiment of the present application;

[0139] Input / output interface 703, used to implement information input and output;

[0140] Communication interface 704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0141] A bus 705 that transmits information between the various components of the device (e.g., the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);

[0142] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0143] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned target detection method based on multi-sensor fusion is implemented.

[0144] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0145] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0147] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0148] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0149] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0150] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0151] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0152] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0153] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0156] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A target detection method based on multi-sensor fusion, characterized in that: The method comprises: Acquire radar point cloud data and multi-view image data of the vehicle's surrounding environment, wherein the multi-view image data is acquired by collecting data from a plurality of camera devices arranged on the vehicle in different directions; Performing perspective division and feature extraction on the radar point cloud data to obtain multi-perspective radar point cloud features and multi-perspective radar point cloud semantic features; Combining the multi-view radar point cloud features and the multiple camera parameter matrices corresponding to the multiple camera devices, performing feature extraction and depth prediction on the multi-view image data to obtain multi-view image semantic features and multi-view image depth information; According to the multi-view image depth information, the multi-view image semantic features and the multi-view radar point cloud semantic features are fused to obtain a multimodal fusion feature under the bird's-eye view perspective; Perform target detection on the multimodal fusion features to obtain a target detection result.

2. The target detection method based on multi-sensor fusion according to claim 1, characterized in that: The performing perspective division and feature extraction on the radar point cloud data to obtain multi-perspective radar point cloud features and multi-perspective radar point cloud semantic features comprises: Dividing the radar point cloud data according to the multiple field of view ranges corresponding to the multiple camera devices to obtain multi-view radar point cloud data; Performing voxel feature extraction and fusion on the multi-view radar point cloud data to obtain multi-view radar columnar point cloud features; Convert the multi-view radar columnar point cloud features to a bird's-eye view to perform multi-scale feature extraction and fusion to obtain the multi-view radar point cloud features; Semantic features are extracted from the multi-view radar point cloud features to obtain semantic features of the multi-view radar point cloud.

3. The target detection method based on multi-sensor fusion according to claim 1, characterized in that: The combining of the multi-view radar point cloud features and the multiple camera parameter matrices corresponding to the multiple camera devices to extract features and predict depth of the multi-view image data to obtain multi-view image semantic features and multi-view image depth information includes: Performing multi-scale feature extraction and fusion on the multi-view image data to obtain multi-view image features; Performing gradient extraction on the multi-view image features to obtain multi-view gradient information; According to the multi-view gradient information and the multi-view radar point cloud features, edge enhancement is performed on the multi-view image features to obtain multi-view image enhancement features; In combination with a plurality of camera parameter matrices corresponding to the plurality of camera devices, semantic feature extraction and depth prediction are performed on the multi-view image enhancement features to obtain the multi-view image semantic features and the multi-view image depth information.

4. The target detection method based on multi-sensor fusion according to claim 3 is characterized in that: The multi-view gradient information includes a multi-view horizontal gradient value and a multi-view vertical gradient value; the edge enhancement of the multi-view image feature is performed according to the multi-view gradient information and the multi-view radar point cloud feature to obtain the multi-view image enhancement feature, which includes: Performing preliminary edge enhancement on the multi-view image features according to the multi-view horizontal gradient values ​​and the multi-view vertical gradient values ​​to obtain preliminary enhanced multi-view image features; The multi-view image preliminary enhancement features and the multi-view radar point cloud features are cross-attentively fused to obtain the multi-view image enhancement features.

5. The target detection method based on multi-sensor fusion according to claim 1, characterized in that: The step of fusing the multi-view image semantic features and the multi-view radar point cloud semantic features according to the multi-view image depth information to obtain a multimodal fusion feature at a bird's-eye view perspective includes: According to the multi-view image depth information, dimensionally expanding the multi-view image semantic features; Performing perspective feature fusion on the multi-view image semantic features after dimension expansion and the multi-view radar point cloud semantic features to obtain corresponding image bird's-eye view features and point cloud bird's-eye view features; A multi-feature fusion network is used to perform multi-scale feature extraction and fusion on the image bird's-eye view features and the point cloud bird's-eye view features to obtain the multimodal fusion features.

6. The target detection method based on multi-sensor fusion according to claim 5, characterized in that: The multi-feature fusion network includes a multi-feature cascade module and a feature fusion module, and the feature fusion module includes a plurality of stacked improved transformers, each of which is obtained by replacing the multi-head self-attention layer set in the basic transformer with a multi-scale deformable large core attention layer; The multi-scale feature extraction and fusion of the image bird's-eye view features and the point cloud bird's-eye view features using a multi-feature fusion network to obtain the multimodal fusion features includes: The image bird's-eye view features and the point cloud bird's-eye view features are cascaded and channel-mapped for dimensionality reduction by the multi-feature cascade module to obtain a multimodal cascade feature; The multi-scale feature extraction and fusion of the multi-modal cascade features are performed through the feature fusion module to obtain the multi-modal fusion features.

7. The target detection method based on multi-sensor fusion according to claim 6, characterized in that: The multi-scale deformable large core attention layer includes a first convolution layer, a second convolution layer, and a plurality of deformable large core attention layers correspondingly arranged at multiple scales; When the multi-scale deformable large core attention layer receives an input feature, the input feature is dimensionally expanded by the first convolutional layer to obtain a plurality of initial features corresponding to the plurality of scales; Extracting the initial features corresponding to the same scale through each deformable large core attention layer, and obtaining the intermediate features corresponding to the same scale; The second convolutional layer is used to fuse the multiple intermediate features corresponding to the multiple scales output by the multiple deformable large core attention layers to obtain output features.

8. A target detection system based on multi-sensor fusion, characterized in that: The system comprises: An acquisition module, used to acquire radar point cloud data and multi-view image data of the vehicle's surrounding environment, wherein the multi-view image data is acquired by collecting multiple camera devices arranged on the vehicle in different directions; A first processing module is used to perform perspective division and feature extraction on the radar point cloud data to obtain multi-perspective radar point cloud features and multi-perspective radar point cloud semantic features; A second processing module is used to combine the multi-view radar point cloud features and the multiple camera parameter matrices corresponding to the multiple camera devices to perform feature extraction and depth prediction on the multi-view image data to obtain multi-view image semantic features and multi-view image depth information; A third processing module is used to fuse the multi-view image semantic features and the multi-view radar point cloud semantic features according to the multi-view image depth information to obtain a multimodal fusion feature under the bird's-eye view perspective; The detection module is used to perform target detection on the multimodal fusion features to obtain a target detection result.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images