Detection method and system based on weak projection mapping and dual-modal sequential enhancement fusion

Abstract

The present invention belongs to the field of computer technology and image processing, and provides a detection method and system based on weak projection mapping and bimodal sequential enhancement fusion, which respectively extracts features from image information and point cloud information, preliminarily aligns image features and point cloud features, and captures the local correlation between point cloud and image; maps point cloud features, calculates attention weights, and uses attention weights to perform weighted summation on value vectors; aggregates feature information of K neighbors of fused features to obtain point-graph aggregate features; maps point-graph aggregate features, point features and point coordinate information to the same channel dimension, and determines the fusion attention gate weights of point-graph aggregate features and point features; obtains fused image features and fused point features according to the fusion attention gate weights; the present invention not only realizes the multi-scale full interaction of the two modalities, but also effectively reduces the propagation of noise and invalid information.

Description

Technical Field

[0001] The present invention belongs to the fields of computer technology and image processing, and in particular relates to a detection method and system based on weak projection mapping and dual-modal sequential enhancement fusion. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Using information fusion from multiple sensors is an important way to achieve robustness and accuracy in 3D object detection. Single sensor devices tend to output low-quality perception information under harsh conditions, such as when the camera encounters poor lighting or occlusion, or when the lidar encounters extreme weather (rain and snow). Especially in low-light conditions, the camera is prone to underexposure, resulting in missing image information; and under the interference of strong light sources such as street lights and oncoming vehicle lights, the camera may be overexposed. These factors will lead to a decrease in image quality, which in turn affects the accuracy of vehicle detection.

[0004] Most of the current mainstream multimodal fusion methods study the integration of two modalities at different stages, including early fusion, deep fusion, and late fusion. Although these methods are effective, when low-quality modal data is input, the early fusion method fuses the sensor data at an early stage, which may introduce unnecessary noise and background interference information during the fusion process, thereby affecting the detection effect; while the late fusion method fuses after independently processing the information of each modality, which may lead to excessive dependence on a single modality and also put forward higher requirements for the detection channels of the two modalities. The feature fusion alignment method has greater potential when facing the suppression of single-modal data.

[0005] However, achieving efficient feature alignment and deep fusion often relies on accurate multi-sensor temporal and spatial calibration, as well as projection-based hard feature mapping methods. This alignment error may have a significant impact in small-scale and long-distance detection, weakening the fusion effect. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a detection method and system based on weak projection mapping and dual-modal sequential enhancement fusion. The present invention adopts a multi-scale sequential enhancement method of point cloud and image features, and adopts a two-stage strategy of non-projection mapping-feature enhancement during fusion; it not only realizes the multi-scale full interaction of the two modalities, but also effectively reduces the propagation of noise and invalid information.

[0007] According to some embodiments, the present invention adopts the following technical solutions:

[0008] A detection method based on weak projection mapping and bimodal sequential enhancement fusion, comprising the following steps:

[0009] Obtain image information and point cloud information;

[0010] Feature extraction is performed on the image information and point cloud information respectively. Based on the extracted point cloud features, image features and point cloud feature coordinates, a neighbor index matrix is ​​generated through the K nearest neighbor algorithm to preliminarily align the image features and point cloud features and capture the local correlation between the point cloud and the image.

[0011] Mapping point cloud features into query vectors, mapping image features into key vectors and value vectors, calculating a similarity matrix between the query vector and the key vector, obtaining an attention weight, and performing weighted summation on the value vector using the attention weight;

[0012] Aggregate the feature information of K neighbors of the fused features to obtain point-graph aggregate features;

[0013] Map the point-image aggregation features, point features and point coordinate information to the same channel dimension, and determine the fusion attention gate weights of the point-image aggregation features and point features;

[0014] According to the fused attention gate weights, the fused image features and fused point features are obtained respectively;

[0015] The fused image features and point features are downsampled, decoded and ROI proposals are generated to achieve target detection.

[0016] As an optional implementation, the following steps are also included: calculating the difference in confidence distribution between the point cloud and the image modality during the ROI proposal generation process to capture the deviation of each modality from the global average confidence, and adaptively weighting and calculating the loss coefficient according to the difference size;

[0017] Perform object detection based on the generated ROI.

[0018] As an optional implementation, the process of generating a neighbor index matrix by using a K-nearest neighbor algorithm and preliminarily aligning image features with point cloud features includes:

[0019] Based on the extracted image features F(B, Ci, H, W) and the aggregated point features F(B, N, Cp) and the point feature coordinates (N, 3), the K nearest neighbor algorithm is used to calculate the K nearest neighbor points of each point in the point cloud in its three-dimensional space, and a neighbor index matrix (N, K) is generated. This matrix is ​​used to capture the geometric structure information of the local area in the point cloud;

[0020] Flatten the input image features into a one-dimensional sequence (B, H*W, Ci), and extract the image features (B, N, Ci) corresponding to the point cloud features from the flattened image features through the projection index of the point cloud in the image space, so as to achieve preliminary alignment of the two modalities in space;

[0021] Among them, B is Batchsize, which indicates the batch size. In deep learning, multiple samples are usually processed at a time. B represents the number of samples in a batch. H (Height) is the height of the image, which indicates the number of vertical pixels of the input image. W (Width) is the width of the image, which indicates the number of horizontal pixels of the input image. K represents the number of neighboring points in the set three-dimensional space, N represents the number of points, and Cp and Ci represent the number of feature channels of points and images respectively.

[0022] As an optional implementation, the process of capturing the local correlation between the point cloud and the image includes using a neighbor index matrix to map the image depth features F i The index is expanded to the dimension matching the K neighbor points to obtain the image depth feature (B, N, K, Ci). At the same time, the point cloud depth feature (B, N, Cp) is copied K times to obtain the corresponding feature expression (B, N, K, Cp).

[0023] As an optional implementation, the neighbor vector L2 norm is performed on the fused features, and the feature information of K neighbors is aggregated to obtain the final mapping feature (B, N, Cip), where Cip represents the number of feature channels of the obtained final mapping feature.

[0024] As an optional implementation, the process of determining the fusion attention gate weight of the point-graph aggregate feature and the point feature includes: multiplying the point-graph aggregate feature with the feature after the point coordinate passes through the fully connected layer, and adding it to the point feature to form a feature representation;

[0025] The fully connected layer and sigmoid activation function are used to compress the weight map to obtain the fusion attention gate weights of the point-map aggregation features and the point features.

[0026] As an optional implementation, the input point feature is multiplied by the fusion attention gate weight to obtain more stable point feature information, and then the adjacent point features are bilinearly interpolated to restore the feature expression similar to the image, and then cascaded with the input image feature to obtain the fused image feature F i .

[0027] As an optional implementation, the fused attention gate weight is multiplied by the point-graph aggregation feature and then cascaded with the point feature to obtain the fused point feature.

[0028] A detection system based on weak projection mapping and bimodal sequential enhancement fusion, comprising:

[0029] A data acquisition module is configured to acquire image information and point cloud information;

[0030] A feature alignment module is configured to extract features from the image information and the point cloud information respectively, generate a neighbor index matrix through a K-nearest neighbor algorithm based on the extracted point cloud features, image features and point cloud feature coordinates, preliminarily align the image features and point cloud features, and capture the local correlation between the point cloud and the image;

[0031] A feature mapping module is configured to map point cloud features into a query vector, map image features into a key vector and a value vector, calculate a similarity matrix between the query vector and the key vector, obtain an attention weight, and perform a weighted summation on the value vector using the attention weight;

[0032] The feature fusion module is configured to aggregate the feature information of K neighbors of the fused feature to obtain the point-graph aggregate feature; map the point-graph aggregate feature, the point feature and the point coordinate information to the same channel dimension, and determine the fusion attention gate weight of the point-graph aggregate feature and the point feature;

[0033] The feature enhancement module is configured to obtain fused image features and fused point features respectively according to the fused attention gate weights;

[0034] The post-processing module is configured to downsample, decode and generate ROI proposals for the fused image features and point features respectively to achieve target detection.

[0035] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps in the above method are completed.

[0036] An electronic device comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the above method are completed.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The present invention adopts a multi-scale sequential enhancement method of point cloud and image features, and adopts a two-stage strategy of non-projection mapping-feature enhancement during fusion, which not only realizes the multi-scale full interaction of the two modalities, but also effectively reduces the propagation of noise and invalid information.

[0039] In the weak projection mapping stage, the present invention explores the relationship between point cloud and image features through the KNN method, and utilizes the Transformer-based cross-attention mechanism to mine the semantic relationship between point space and pixel features, thereby promoting information propagation between point cloud and image features and forming a consistent feature representation.

[0040] In the feature enhancement stage, the present invention enhances the expression of reliable features by introducing attention gate weights. At the same time, when constructing the weights, the spatial position information is incorporated to enhance the model's spatial understanding of 2D features, thereby promoting a closer integration between point clouds and images.

[0041] Aiming at the scenes with low light or tiny or occluded targets, where there is often a large difference in the confidence scores of point clouds and images, the present invention proposes an adaptive multimodal consistency loss to more reasonably use the confidence scores under different modes in the model training process, thereby making the final prediction results more accurate and reasonable.

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0044] Figure 1 is a schematic diagram of a dual-stream fusion encoding process according to an embodiment;

[0045] Figure 2 is a schematic diagram of a decoder and ROI region generation process according to an embodiment;

[0046] Figure 3 is a schematic diagram of a weak projection mapping process of an embodiment;

[0047] Figure 4 It is a schematic diagram of a PPI and IPP enhancement module of an embodiment. DETAILED DESCRIPTION

[0048] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0049] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0050] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0051] In the absence of conflict, the embodiments in this application and the features in the embodiments may be combined with each other.

[0052] Embodiment 1

[0053] A target detection method such as Figure 1 and Figure 2 As shown in the figure, the method takes the original RGB image and point cloud as input, and inputs two independent feature extraction channels respectively. Each channel contains an independent encoder and decoder for feature extraction. The image encoder and decoder are composed of a simple structure. The receptive field is gradually expanded through four convolution blocks to obtain multi-scale image features, and then the original image resolution is restored with transposed convolution layers of different step lengths. Finally, a feature map containing rich image semantic information is obtained through cascading. The point cloud channel uses PointNet++ as the backbone network. The encoder performs multi-scale point grouping feature aggregation through 4 aggregation abstraction layers (SA), and the decoder is a FP module to realize feature propagation and point resolution recovery, thereby generating richer point features.

[0054] This embodiment mainly proposes a target detection method based on weak projection mapping and bimodal sequential enhancement fusion, and its improvements to the above process mainly lie in two parts.

[0055] The first part is to innovatively design the PPI and IPP fusion module in the encoder or encoding process to better utilize the complementarity between the two modalities. Figure 4 shown.

[0056] Specifically, both modules are divided into two stages. The first stage is the feature alignment and mapping stage, which is called the KNN-based weak projection mapping module, or KAM module for short. Feature alignment and mapping are important components of multimodal fusion methods, which can reduce the difficulty of fusion caused by the heterogeneity between the two modalities.

[0057] like Figure 3As shown in the figure, the module inputs the image features F (B, Ci, H, W) and the aggregated point features F (B, N, Cp) extracted by the corresponding encoder features and the point feature coordinates (N, 3). Among them, B is Batchsize, which indicates the batch size. In deep learning, multiple samples are usually processed at a time. B represents the number of samples contained in a batch, and H (Height) is the height of the image, which indicates the number of vertical pixels of the input image. W (Width) is the width of the image, which indicates the number of horizontal pixels of the input image; N represents the number of points, and Cp and Ci represent the number of feature channels of the point and the image respectively. First, the K nearest neighbors (KNN) algorithm is used to calculate the K nearest neighbors of each point in the point cloud in its three-dimensional space to generate a neighbor index matrix (N, K). This matrix is ​​used to capture the geometric structure information of the local area in the point cloud. Then, the input image features are flattened into a one-dimensional sequence (B, H*W, Ci), and the image features (B, N, Ci) corresponding to the point cloud features are extracted from the flattened image features through the projection index of the point cloud in the image space. This ensures the initial alignment of the two modalities in space, and the generated image features have the same dimension as the point cloud features.

[0058] Use the neighbor index matrix to map the image depth features F i The index is expanded to the dimension matching the K neighbor points to obtain the image deep features (B, N, K, Ci). At the same time, the point cloud deep features are replicated K times to obtain similar feature expressions (B, N, K, Cp). The local correlation between the point cloud and the image is further captured to form a cross-modal neighbor feature representation. Then, through three linear layers, the point cloud features are mapped to query vectors (Q), and the image features are mapped to key vectors (K) and value vectors (V).

[0059] Based on the cross-modal alignment, the similarity matrix between the query vector and the key vector is calculated to obtain the attention weights, and these weights are used to perform weighted summation on the value vector.

[0060] Finally, the L2 norm of the nearest neighbor vector is performed on the fused features, and the feature information of K neighbors is aggregated to obtain the final mapping features (B, N, Cip). The point-graph aggregated features (B, N, Cip) in the mapping stage have similar feature forms to the point features (B, N, Cp), so it is easier to achieve fusion.

[0061] The second stage is the feature fusion and enhancement stage. During the fusion, we hope to establish an effective attention mechanism to achieve reliable feature enhancement while suppressing the information noise introduced by challenging scenarios. Specifically, we input the point-graph aggregation features, point features, and point coordinate information into the fully connected layer to map them to the same channel dimension. Different from the usual simple addition method, we use the property of element-wise multiplication to fuse different subspace features and combine the point-graph aggregation features with the point coordinates F. pc The features after the fully connected layer are multiplied and added to the point features to form a compact feature representation. Then it is compressed into a weight map through the fully connected layer and the sigmoid activation function. So far, we have obtained the fusion attention gate weight w of the point-graph aggregation feature and the point feature. The formula is as follows:

[0062] ;

[0063] Among them, W 1 , W 2 , W 3 , W 4 Respectively represent the learnable weight matrix, F p Represents the point feature tensor, F i represents the point-graph aggregation feature, F pc represents the point coordinate feature tensor, Represents the sigmoid function.

[0064] We divide the output into two different modules, PPI and IPP. The PPI module finally outputs the fused image features, and the IPP module finally outputs the fused point features. The PPI module multiplies the input point features with the attention gate weight w to obtain more stable point feature information, and then sends the adjacent point features to bilinear interpolation to restore the feature expression similar to the image. Then it is cascaded with the input image features to obtain the fused image features F i IPP is similar to PPI. Since the fused attention gate weight itself has an expression similar to the point feature, after multiplying it with the point-graph aggregation feature and concatenating it with the point feature, the fused point feature F can be obtained. p After passing through the IPP and PPI modules in sequence, the enhanced point features and image features will be fed to the next layer of the encoder as the input of the next scale.

[0065] Of course, other processes in the target detection method can adopt existing technologies, such as the decoding process, etc., which will not be described in detail here.

[0066] The second part of the improvement of this embodiment is that in the overall process, the confidence score will directly affect the selection quality of the candidate box and affect the final detection performance. For example, in the detection of long-distance and small targets, the prediction score brought by the dense and rich semantic information of the image is obviously more worthy of attention than radar. When dealing with occluded objects or low-light environments, the boundary distinction ability and noise resistance ability from geometric information are more reliable.

[0067] This embodiment uses adaptive multimodal consistency loss (D_MC) to make full use of the differences and consistency. Specifically, given the target point confidence score S p and the corresponding image pixel confidence score S i , calculate the average value S a D_MC quantifies the difference in confidence distribution between point cloud and image modalities by calculating KL divergence to capture the deviation of each modality from the global average confidence and obtain more fine-grained information. Then, we formulate the following strategy: First, when the confidence score S p With S i Any one less than the threshold score t 1 , it is considered not to be a target point and is not considered. p With S i The difference is greater than t 2 When , one of the modes has a higher confidence score, and the loss coefficient is obtained in an adaptive weighted manner. Finally, when S p With S i When the scores are not much different, the two loss coefficients are balanced by a fixed coefficient.

[0068] The loss D_MC is formulated as follows:

[0069] ;

[0070] ;

[0071] .

[0072] in, are the loss coefficient, L dmc is the final loss, KL() is the (Kullback-LeiblerDivergence) divergence calculation, which is an asymmetric measure used to measure the difference between two probability distributions, indicating the information loss or additional overhead from one distribution to another; N is the total number of input points, I m It is an indicator function, the output 1 represents foreground information, and the output 0 represents background information.

[0073] Embodiment 2

[0074] A detection system based on weak projection mapping and bimodal sequential enhancement fusion, comprising:

[0075] A data acquisition module is configured to acquire image information and point cloud information;

[0076] A feature alignment module is configured to extract features from the image information and the point cloud information respectively, generate a neighbor index matrix through a K-nearest neighbor algorithm based on the extracted point cloud features, image features and point cloud feature coordinates, preliminarily align the image features and point cloud features, and capture the local correlation between the point cloud and the image;

[0077] A feature mapping module is configured to map point cloud features into a query vector, map image features into a key vector and a value vector, calculate a similarity matrix between the query vector and the key vector, obtain an attention weight, and perform a weighted summation on the value vector using the attention weight;

[0078] The feature fusion module is configured to aggregate the feature information of K neighbors of the fused feature to obtain the point-graph aggregate feature; map the point-graph aggregate feature, the point feature and the point coordinate information to the same channel dimension, and determine the fusion attention gate weight of the point-graph aggregate feature and the point feature;

[0079] The feature enhancement module is configured to obtain fused image features and fused point features respectively according to the fused attention gate weights;

[0080] The post-processing module is configured to downsample, decode and generate ROI proposals for the fused image features and point features respectively to achieve target detection.

[0081] Embodiment 3

[0082] The present implementation provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the following process:

[0083] Obtain image information and point cloud information;

[0084] Feature extraction is performed on the image information and point cloud information respectively. Based on the extracted point cloud features, image features and point cloud feature coordinates, a neighbor index matrix is ​​generated through the K nearest neighbor algorithm to preliminarily align the image features and point cloud features and capture the local correlation between the point cloud and the image.

[0085] Mapping point cloud features into query vectors, mapping image features into key vectors and value vectors, calculating a similarity matrix between the query vector and the key vector, obtaining an attention weight, and performing weighted summation on the value vector using the attention weight;

[0086] Aggregate the feature information of K neighbors of the fused features to obtain point-graph aggregate features;

[0087] Map the point-image aggregation features, point features and point coordinate information to the same channel dimension, and determine the fusion attention gate weights of the point-image aggregation features and point features;

[0088] According to the fused attention gate weights, the fused image features and fused point features are obtained respectively;

[0089] The fused image features and point features are downsampled, decoded and ROI proposals are generated to achieve target detection.

[0090] Or the steps provided in Example 1.

[0091] Embodiment 4:

[0092] This implementation provides a computer-readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer-readable storage medium provides a storage space that stores the processing system of the electronic device.

[0093] In addition, the storage space also stores one or more instructions suitable for being loaded and executed by the processor, and these instructions may be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here may be a high-speed RAM memory, or a non-volatile memory, such as at least one disk memory; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor.

[0094] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to implement the following process:

[0095] Obtain image information and point cloud information;

[0096] Feature extraction is performed on the image information and point cloud information respectively. Based on the extracted point cloud features, image features and point cloud feature coordinates, a neighbor index matrix is ​​generated through the K nearest neighbor algorithm to preliminarily align the image features and point cloud features and capture the local correlation between the point cloud and the image.

[0097] Mapping point cloud features into query vectors, mapping image features into key vectors and value vectors, calculating a similarity matrix between the query vector and the key vector, obtaining an attention weight, and performing weighted summation on the value vector using the attention weight;

[0098] Aggregate the feature information of K neighbors of the fused features to obtain point-graph aggregate features;

[0099] Map the point-image aggregation features, point features and point coordinate information to the same channel dimension, and determine the fusion attention gate weights of the point-image aggregation features and point features;

[0100] According to the fused attention gate weights, the fused image features and fused point features are obtained respectively;

[0101] The fused image features and point features are downsampled, decoded and ROI proposals are generated to achieve target detection.

[0102] Or the steps provided in Example 1.

[0103] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of an all-hardware embodiment, an all-software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of one or more computer-usable storage media (including but not limited to disk storage, CD - ROM , optical storage, etc.).

[0104] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0105] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0107] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principle of the present invention without creative labor shall be included in the protection scope of the present invention.

Claims

1. A detection method based on weak projection mapping and bimodal sequential enhancement fusion, characterized in that: The following steps are involved: Obtain image information and point cloud information; Feature extraction is performed on the image information and point cloud information respectively. Based on the extracted point cloud features, image features and point cloud feature coordinates, a neighbor index matrix is ​​generated through the K nearest neighbor algorithm to preliminarily align the image features and point cloud features and capture the local correlation between the point cloud and the image. Mapping point cloud features into query vectors, mapping image features into key vectors and value vectors, calculating a similarity matrix between the query vector and the key vector, obtaining an attention weight, and performing weighted summation on the value vector using the attention weight; Aggregate the feature information of K neighbors of the fused features to obtain point-graph aggregate features; Map the point-image aggregation features, point features and point coordinate information to the same channel dimension, and determine the fusion attention gate weights of the point-image aggregation features and point features; According to the fused attention gate weights, the fused image features and fused point features are obtained respectively; The fused image features and point features are downsampled, decoded, and ROI proposals are generated to achieve target detection; The process of generating a neighbor index matrix through the K nearest neighbor algorithm and preliminarily aligning the image features and point cloud features includes: Based on the extracted image features F(B, Ci, H, W) and the aggregated point features F(B, N, Cp) and the point feature coordinates (N, 3), the K nearest neighbor algorithm is used to calculate the K nearest neighbor points of each point in the point cloud in its three-dimensional space, and a neighbor index matrix (N, K) is generated. This matrix is ​​used to capture the geometric structure information of the local area in the point cloud; Flatten the input image features into a one-dimensional sequence (B, H*W, Ci), and extract the image features (B, N, Ci) corresponding to the point cloud features from the flattened image features through the projection index of the point cloud in the image space, so as to achieve preliminary alignment of the two modalities in space; Where B is the batch size, H is the height of the image, which indicates the number of vertical pixels of the input image, and W is the width of the image, which indicates the number of horizontal pixels of the input image; N represents the number of points, and Cp and Ci represent the number of feature channels of points and images respectively; The process of capturing local correlations between point clouds and images involves mapping image deep features F with a neighbor index matrix i The index is expanded to the dimension matching the K neighboring points to obtain the image depth feature (B, N, K, Ci), and the point cloud depth feature is replicated K times to obtain the corresponding feature expression (B, N, K, Cp); Perform the L2 norm of the neighbor vector on the fused features, aggregate the feature information of K neighbors, and obtain the final mapping features (B, N, Cip), where Cip represents the number of feature channels of the final mapping features obtained; The process of determining the fusion attention gate weight of the point-graph aggregate feature and the point feature includes: multiplying the point-graph aggregate feature with the feature after the point coordinate passes through the fully connected layer, and adding it to the point feature to form a feature representation; The weight map is compressed into a weight map through the fully connected layer and the sigmoid activation function to obtain the fusion attention gate weights of the point-map aggregation features and the point features; The input point feature is multiplied by the fusion attention gate weight to obtain more stable point feature information, and then the adjacent point features are sent to bilinear interpolation to restore the feature expression similar to the image, and then cascaded with the input image feature to obtain the fused image feature; The fused attention gate weight is multiplied by the point-graph aggregation feature and then concatenated with the point feature to obtain the fused point feature.

2. The detection method based on weak projection mapping and bimodal sequential enhancement fusion as claimed in claim 1, characterized in that: The following steps are also included: In the process of ROI proposal generation, the difference in confidence distribution between point cloud and image modalities is calculated to capture the deviation of each modality from the global average confidence, and the loss coefficient is calculated by adaptive weighting according to the difference size; Perform object detection based on the generated ROI.

3. A detection system based on weak projection mapping and bimodal sequential enhancement fusion, characterized in that: include: A data acquisition module is configured to acquire image information and point cloud information; A feature alignment module is configured to extract features from the image information and the point cloud information respectively, generate a neighbor index matrix through a K-nearest neighbor algorithm based on the extracted point cloud features, image features and point cloud feature coordinates, preliminarily align the image features and point cloud features, and capture the local correlation between the point cloud and the image; A feature mapping module is configured to map point cloud features into a query vector, map image features into a key vector and a value vector, calculate a similarity matrix between the query vector and the key vector, obtain an attention weight, and perform a weighted summation on the value vector using the attention weight; The feature fusion module is configured to aggregate the feature information of K neighbors of the fused feature to obtain the point-graph aggregate feature; map the point-graph aggregate feature, the point feature and the point coordinate information to the same channel dimension, and determine the fusion attention gate weight of the point-graph aggregate feature and the point feature; The feature enhancement module is configured to obtain fused image features and fused point features respectively according to the fused attention gate weights; The post-processing module is configured to downsample, decode and generate ROI proposals for the fused image features and point features to achieve target detection; The process of generating a neighbor index matrix through the K nearest neighbor algorithm and preliminarily aligning the image features and point cloud features includes: Based on the extracted image features F(B, Ci, H, W) and the aggregated point features F(B, N, Cp) and the point feature coordinates (N, 3), the K nearest neighbor algorithm is used to calculate the K nearest neighbor points of each point in the point cloud in its three-dimensional space, and a neighbor index matrix (N, K) is generated. This matrix is ​​used to capture the geometric structure information of the local area in the point cloud; Flatten the input image features into a one-dimensional sequence (B, H*W, Ci), and extract the image features (B, N, Ci) corresponding to the point cloud features from the flattened image features through the projection index of the point cloud in the image space, so as to achieve preliminary alignment of the two modalities in space; Where B is the batch size, H is the height of the image, which indicates the number of vertical pixels of the input image, and W is the width of the image, which indicates the number of horizontal pixels of the input image; N represents the number of points, and Cp and Ci represent the number of feature channels of points and images respectively; The process of capturing local correlations between point clouds and images involves mapping image deep features F with a neighbor index matrix i The index is expanded to the dimension matching the K neighboring points to obtain the image depth feature (B, N, K, Ci), and the point cloud depth feature is replicated K times to obtain the corresponding feature expression (B, N, K, Cp); Perform the L2 norm of the neighbor vector on the fused features, aggregate the feature information of K neighbors, and obtain the final mapping features (B, N, Cip), where Cip represents the number of feature channels of the final mapping features obtained; The process of determining the fusion attention gate weight of the point-graph aggregate feature and the point feature includes: multiplying the point-graph aggregate feature with the feature after the point coordinate passes through the fully connected layer, and adding it to the point feature to form a feature representation; The weight map is compressed into a weight map through the fully connected layer and the sigmoid activation function to obtain the fusion attention gate weights of the point-map aggregation features and the point features; The input point feature is multiplied by the fusion attention gate weight to obtain more stable point feature information, and then the adjacent point features are sent to bilinear interpolation to restore the feature expression similar to the image, and then cascaded with the input image feature to obtain the fused image feature; The fused attention gate weight is multiplied by the point-graph aggregation feature and then concatenated with the point feature to obtain the fused point feature.

4. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of the method according to any one of claims 1 to 2.

5. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps in the method according to any one of claims 1 to 2 are completed.

Images