A multi-modal feature fusion method based on a vehicle-mounted camera

Abstract

The application discloses a kind of based on vehicle camera multimodal feature fusion method, including step 1: feature attention stage: constructing grouping feature cross attention mechanism network, cross positioning spatial information to feature map, extract key features, obtain feature attention diagram, step 2: feature fusion stage: construct dynamic feature redefinition network, input feature attention diagram to dynamic feature redefinition network, obtain similarity score according to the correlation of input feature map, and the corresponding features between modes are adaptively weighted to obtain redefined feature diagram;Cross-modal feature extraction network is constructed, and the redefined feature diagram is input to cross-modal feature extraction network, the feature information of another mode is fused, and the output of next stage under double branch network is obtained.The multimodal fusion method of the application can better utilize the multimodal feature information of vehicle camera, so that the feature extraction network effectively learns the feature information of target.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer image processing, and particularly relates to a multi-modal feature fusion method based on a vehicle-mounted camera. BACKGROUND

[0002] In the field of automatic driving, with the proliferation of modular sensors in cars, multi-modal fusion for semantic segmentation has recently made rapid progress, which in turn has also aroused people's interest in assembling more and more sensors to achieve higher and higher segmentation accuracy to strengthen the stability of automatic driving. The fusion of vehicle-mounted multi-modal cameras such as depth, multi-view, event and RGB pictures, and laser radar can improve the robustness of semantic segmentation, such as when the visual camera encounters insufficient light or fog, infrared information can provide useful complementary clues. However, how to effectively fuse RGB images and other modal data is still a pending challenge. Previous work adopts a dual-end parallel network to perform feature extraction on RGB and RGBX (representing features of other modalities) respectively, and the information between modalities adopts a simple fusion strategy, such as merging them at the input; connecting multi-modal features within the model, or applying attention to each data modality. These fusion strategies are straightforward, but the utilization of modal information is not sufficient.

[0003] The current feature fusion method between modalities is divided into early fusion, deep fusion, late fusion and asymmetric fusion according to the stage of fusion. Early fusion is to fuse the data between each modality before feature extraction, and then perform feature extraction through the network. Early fusion is easy to cause semantic information asymmetry between modalities and produce redundant information; deep fusion refers to the fusion of features generated in the network; late fusion is the fusion of feature maps after the feature extraction of two modalities; late fusion reduces the correlation between modalities; in asymmetric fusion, at least one branch dominates, and the other branch provides auxiliary information to perform the final task, but asymmetric fusion only has the dominance from one branch, reducing the importance of the other modality. SUMMARY

[0004] The application proposes a multi-modal feature fusion method based on a vehicle-mounted camera to strengthen the fusion of complementary information of modalities and reduce the noise of fusion between different modalities. It contains a grouping cross attention module GCA and a cross-modal explicit attention enhanced fusion module FFM. The GCA module is used to enhance the positioning and extraction ability of important features of the input feature map. The FFM module can perform more effective cross-fusion of inter-modal information.

[0005] To solve the above technical problems, the application provides the following technical scheme: a multi-modal feature fusion method based on a vehicle-mounted camera, comprising the following steps:

[0006] S1, feature attention stage: a grouped feature cross attention mechanism network is constructed, an RGBX picture generated by an RGB camera and a vehicle-mounted camera is input into a double-branch feature extraction network to obtain output feature maps F_RGB and F_RGBX of the i-th stage, and then the feature maps F_RGB and F_RGBX are input into the grouped feature cross attention mechanism network, spatial information of the two feature maps is cross-positioned, key features are extracted, and feature attention maps RGB_Atten and RGBX_Atten are obtained. S2, feature fusion stage: comprising: S2-1, constructing a dynamic feature redefinition network, inputting the feature attention maps RGB_Atten and RGBX_Atten into the dynamic feature redefinition network, obtaining a similarity score through the correlation of the feature attention maps RGB_Atten and RGBX_Atten, and adaptively weighting the corresponding features between the modes. Redefined feature maps R_RGB and R_RGBX are obtained.

[0007] Step S2-2, constructing a cross-modal feature extraction network, inputting the redefined feature maps R_RGB and R_RGBX into the cross-modal feature extraction network, fusing feature information of another mode, and obtaining final modal fusion feature maps Fuse_RGB and Fuse_RGBX, which are used to input into a double-branch feature extraction network of the i+1-th stage to further extract features.

[0008] Further, the foregoing step S1 comprises the following substeps:

[0009] S101, using a reshape operation to change F_RGB, F_RGBX e R Ci×Hi×Wi into R G×Hi×Wi ;

[0010] Wherein, Ci, Hi, Wi are respectively the channel number, the length of the feature map and the width of the feature map of the output feature map of the i-th stage, and G is the number of groups of the feature map;

[0011] S102, inputting the grouped feature map into an HW feature compression attention network to obtain a spatial attention code containing H and W two-dimensional features as follows:

[0012] RGB_HWAtten=Se_HW(F_RGB),

[0013] RGBX_HWAtten=Se_HW(F_RGBX);

[0014] Wherein, RGB_HWAtten is an attention code generated by the F_RGB through the HW feature compression attention network, and Se_HW function represents an operation of compressing and extracting H and W two-dimensional features;

[0015] S103, then the attention encoding containing spatial coordinate information is aggregated through the channel information of the spatial attention network feature map to obtain feature attention maps RGB_Atten and RGBX_Atten, as follows:

[0016] RGB_Atten = Spatial(RGB_HWAtten)

[0017] RGBX_Atten = Spatial(RGBX_HWAtten)

[0018] wherein the Spatial function represents a further extraction operation of spatial attention on the attention encoding Se_Atten, and finally restores the grouping RGB_Atten ∈ R Ci×Hi×Wi .

[0019] Further, the aforementioned step S102 includes the following sub-steps:

[0020] S102-1, the feature maps RGBh_Atten and RGBXh_Atten containing W spatial coordinate information are aggregated along the H direction through one-dimensional pooling operation, as follows:

[0021]

[0022]

[0023] wherein c represents one channel of the input feature map, and the W spatial information is compressed and encoded to obtain RGBh_Atten; h represents the height of a certain channel of the input feature map, w represents the width of a certain channel of the input feature map, and j represents a variable for traversing w pixels;

[0024] S102-2, the 3x3 empty convolution encoding decoding structure is used, and after the h_swish activation function, it is multiplied with the input to obtain the RGB_HAtten containing W spatial information, as follows:

[0025] RGB_HAtten = h_swish(con3x3(RGBh_Atten))*F_RGB

[0026] RGBX_HAtten = h_swish(con3x3(RGBXh_Atten))*F_RGBX

[0027] S102-3, the RGB_HAtten and RGBX_HAtten containing W spatial information are aggregated through one-dimensional pooling operation along the W direction to obtain the feature maps RGBw_Atten and RGBXw_Atten containing H spatial coordinate information, as follows:

[0028]

[0029]

[0030] wherein w represents the width of input features Figure One of the channel, and j represents a variable traversing h pixels;

[0031] S102-4, an encoding-decoding structure of the input 3x3 hole convolution is multiplied with the input after passing through the h_swish activation function to obtain the RGB_HWAtten containing the HW spatial information, as follows:

[0032] RGB_HWAtten = h_swish(con3x3(RGBw_Atten))*RGB_HAtten

[0033] RGBX_HWAtten = h_swish(con3x3(RGBXw_Atten))*RGBX_HAtten.

[0034] Further, the aforementioned step S103 is specifically, the spatial attention network includes a 3x3 convolution operation to further converge spatial information, and then the mean variance standardization operation is generated to generate a spatial attention weight map, and finally multiplied with the input element by element to obtain the final RGB_Atten and RGBX_Atten feature attention map, as follows:

[0035] RGB_Atten = Norm(Con3x3(RGB_HWAtten))*RGB_HWAtten

[0036] RGBX_Atten = Norm(Con3x3(RGBX_HWAtten))*RGBX_HWAtten

[0037] wherein Norm represents the mean normalization operation of the attention map after convolution.

[0038] Further, the aforementioned step S2-1 includes the following substeps:

[0039] S2-1.1, the dynamic feature redefinition network includes a 3X3 convolution network for reducing the channel of the RGB_Atten and RGBX_Atten attention feature map in the previous stage to 1, and then flattening as follows:

[0040] F_RGB_Atten = Flatten(Con3x3(RGB_Atten))

[0041] F_RGBX_Atten = Flatten(Con3x3(RGBX_Atten))

[0042] wherein, Con3X3 represents a 3x3 convolution operation, Flatten represents a flattening operation, F_RGB_Atten∈R 1×HiWi denotes a flattened feature sequence;

[0043] S2-1.2, obtain a similarity score by calculating the cosine similarity of the two flattened feature sequences, multiply the similarity score with the input modal feature, obtain the redefined feature maps R_RGB and R_RGBX, as follows:

[0044] R_RGB = RGB_Atten * Cos_Simlar(F_RGB_Atten, F_RGBX_Atten)

[0045] R_RGBX = RGBX_Atten * Cos_Simlar(F_RGB_Atten, F_RGBX_Atten)

[0046] wherein, Cos_Simlar represents calculating the cosine similarity score of two sequences.

[0047] Further, the aforementioned step S2-2 is specifically: input the redefined feature maps R_RGB and R_RGBX into the cross-modal feature extraction network, obtain the cross-modal global attention by calculating Cross self_attention, obtain the final modal fusion feature maps Fuse_RGB and Fuse_RGBX as follows:

[0048] Fuse_RGB, Fuse_RGBX = Cross self_attention(R_RGB, R_RGBX)

[0049] wherein, Cross self_attention represents a feature information operation of fusing R_RGB and R_RGBX.

[0050] Further, the aforementioned one kind based on vehicle-mounted camera multi-modal feature fusion method, Cross self_attention in cross-modal feature extraction network is based on the improvement of transformer's self_attention, encodes R_RGB, R_RGBX respectively, calculates the corresponding three matrix vectors Q, K, V containing input feature semantic representation, then according to self_attention, respectively calculates global attention map Atten_RGB and Atten_RGBX, multiplies the global attention maps of two modalities to fuse into Atten_fuse, as follows:

[0051]

[0052]

[0053] Atten_fuse=Atten_RGB*Atten_RGBX

[0054] The fused Atten_fuse is multiplied by V of R_RGB respectively, and then V generated by the global attention map Atten_RGB of R_RGB and cross-modal multiplication of R_RGBX is added, cross-modal information is realized, and Fuse_RGB of fused RGBX information and Fuse_RGBX of fused RGB information are obtained.

[0055] Fuse_RGB=Atten_fuse*V RGB +Atten_RGB*V RGBX

[0056] Fuse_RGBX=Atten_fuse*V RGBX +Atten_RGBX*V RGB

[0057] Another aspect of the present application provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method of the present application.

[0058] The present application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the steps of the method of the present application.

[0059] Compared with the prior art, the beneficial technical effects of the above technical solutions of the present application are as follows:

[0060] The multi-modal feature fusion method based on the vehicle-mounted camera provided by the application can better utilize the feature information of the RGBX multi-modal, and enable the feature extraction network to effectively learn the feature information of the target. The method mainly consists of a lightweight attention module GCA and an explicit attention enhancement fusion module FFM. The series one-dimensional global pooling structure of the GCA module can compress the global spatial information while reducing the loss of spatial information, strengthening the connection of spatial information in the horizontal and vertical directions, which is conducive to establishing long-distance spatial information while protecting the information saved in the other direction, better positioning the spatial position information of the feature, thereby improving the learning ability of the network. At the same time, the normalization operation is adopted to prevent the difference from being too large, and the interference of irrelevant information is eliminated to clearly define the target feature. The cosine similarity algorithm of the FFM module adaptively weights the similar features between the modes, calibrates the features between the modes, reduces the noise between the modes, and facilitates the subsequent fusion of the information between the modes. At the same time, the cross-self attention cross-fuses the feature attention maps between the two modes, and strengthens the utilization of the information between the modes. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 The cross-modal fusion structure diagram based on the double-branch semantic segmentation network of the application.

[0062] Figure 2 The attention module GCA structure diagram for grouping cross positioning spatial coordinate information.

[0063] Figure 3 The Se_HW Attention structure diagram.

[0064] Figure 4 The Spatial Attention structure diagram.

[0065] Figure 5 The cross-modal explicit attention enhancement fusion module structure diagram.

[0066] Figure 6 The method flowchart of the application. DETAILED DESCRIPTION

[0067] In order to better understand the technical content of the application, specific embodiments are described below with the aid of the accompanying drawings.

[0068] Aspects of the present application are described herein with reference to the drawings, which are described below. Embodiments of the present application are not limited to the illustrated examples. It should be understood that the present application can be embodied in many different forms and should not be limited to any of the embodiments set forth below. In addition, some aspects of the present application can be used independently of any other aspects of the present application. Further, implementations of the present application can be implemented using any of a wide variety of systems and methods.

[0069] Reference is made to Figure 6 The method flowchart of the present application, with reference to Figure 1 Based on the cross-modal fusion structure diagram of the dual-branch semantic segmentation network, the present application proposes a multi-modal feature fusion method based on vehicle-mounted camera, comprising the following steps:

[0070] S1, feature attention stage: construct a group feature cross attention mechanism network (GCA, Group Coordinate Attention) to input the RGB and vehicle-mounted camera generated RGBX picture into the dual-branch feature extraction network to obtain the output feature map F_RGB, F_RGBX of the i stage, then input the feature map F_RGB, F_RGBX into the group feature cross attention mechanism network, cross locate the spatial information of the two feature maps, extract the key features, and obtain the feature attention map RGB_Atten and RGBX_Atten.

[0071] S2, feature fusion stage: including: S2-1, constructing a dynamic feature redefinition network, inputting the feature attention map RGB_Atten, RGBX_Atten into the dynamic feature redefinition network, obtaining the similarity score through the correlation of the feature attention map RGB_Atten and RGBX_Atten, and adaptively weighting the corresponding features between the modes. Get the redefined feature map R_RGB and R_RGBX;

[0072] Step S2-2, constructing a cross-modal feature extraction network (FFM, Feature Fusion Module), inputting the redefined feature map R_RGB and R_RGBX into the cross-modal feature extraction network, fusing the feature information of another mode, and obtaining the final modal fusion feature map Fuse_RGB and Fuse_RGBX, which are used to input into the dual-branch feature extraction network of the i+1 stage to further extract features.

[0073] As Figure 2As shown, first, the GCA divides the input feature map according to the channel dimension into feature groups representing different semantic features in order to strengthen the learning of specific semantic sub-features and reduce the parameter quantity and computational quantity, so that the subsequent network can learn the sub-features of the input feature map in detail. A one-dimensional global pooling operation is used instead of the two-dimensional global pooling operation commonly used in the previous attention mechanism to avoid the large loss of spatial information caused by the two-dimensional global pooling.

[0074] Further, the step grouping feature cross-attention mechanism network includes a feature grouping module, an HW feature compression attention network and a spatial attention network, and step S1 includes the following sub-steps:

[0075] S101, using reshape operation to change F_RGB, F_RGBX∈R Ci×Hi×Wi to R G×Hi×Wi ;

[0076] Wherein, Ci, Hi, Wi are the channel number, the length of the feature map and the width of the feature map of the i-th stage output feature map respectively, and G is the number of feature map grouping;

[0077] S102, input the grouped feature map into the HW feature compression attention network to obtain the spatial attention coding containing H, W two-dimensional features as follows:

[0078] RGB_HWAtten=Se_HW(F_RGB),

[0079] RGBX_HWAtten=Se_HW(F_RGBX);

[0080] Wherein, RGB_HWAtten is the attention coding generated by F_RGB through the HW feature compression attention network, and Se_HW function represents the operation of compressing and extracting H, W two-dimensional features.

[0081] A serial one-dimensional global pooling structure Se_HW Attention is proposed to encode and extract the spatial position information in each feature group. While reducing the loss of spatial information, the spatial information is aggregated into two feature maps along the vertical and horizontal directions respectively, and further encoded to improve the spatial information expression ability of important information. At the same time, the serial spatial information coding structure can strengthen the connection between spatial dimensions, and the shared coding and decoding structure with 3x3 hollow convolution is introduced in the serial structure to enhance the learning ability of the model to spatial information and better extract the spatial position information of each feature group.

[0082] In combination with Figure 3 , step S102 includes the following sub-steps:

[0083] S102-1, aggregate the feature maps containing W spatial coordinate information RGBh_Atten and RGBXh_Atten along the H direction by one-dimensional pooling operation, as follows:

[0084]

[0085]

[0086] wherein c represents a channel of the input F_RGB, and RGBh_Atten is obtained by compressing and encoding the W spatial information; h represents the height of the input feature Figure One map of one channel, w represents the width of the input feature Figure One map of one channel, and j represents a variable traversing the w pixels.

[0087] S102-2, utilize the encoding-decoding structure of 3x3 hollow convolution, multiply the input after the h_swish activation function to obtain RGB_HAtten containing W spatial information, as follows:

[0088] RGB_HAtten = h_swish(con3x3(RGBh_Atten)) * F_RGB

[0089] RGBX_HAtten = h_swish(con3x3(RGBXh_Atten)) * F_RGBX

[0090] S102-3, aggregate the feature maps containing H spatial coordinate information RGBw_Atten and RGBXw_Atten along the W direction by one-dimensional pooling operation, as follows:

[0091]

[0092]

[0093] wherein w represents the width of the input feature Figure One map of one channel, and j represents a variable traversing the H pixels.

[0094] S102-4, input the encoding-decoding structure of 3x3 hollow convolution, multiply the input after the h_swish activation function to obtain RGB_HWAtten containing HW spatial information, as follows:

[0095] RGB_HWAtten = h_swish(con3x3(RGBw_Atten)) * RGB_HAtten

[0096] RGBX_HWAtten = h_swish(con3x3(RGBXw_Atten)) * RGBX_HAtten.

[0097] S103, then the attention encoding containing spatial coordinate information is aggregated through the channel information of the spatial attention network feature map to obtain feature attention maps RGB_Atten and RGBX_Atten, as follows:

[0098] RGB_Atten = Spatial(RGB_HWAtten)

[0099] RGBX_Atten = Spatial(RGBX_HWAtten)

[0100] Wherein, the Spatial function represents a further extraction operation of spatial attention on the attention encoding Se_Atten, and finally restores the group RGB_Atten ∈ R Ci×Hi×Wi .

[0101] The h_swish nonlinear activation function is used to replace the traditional sigmoid activation function to improve the network convergence speed and gradient disappearance phenomenon, and finally the normalized spatial attention Spatial Attention is proposed to strengthen the information fusion between each channel in the group, eliminate the interference of background information and irrelevant information, gradually pay attention to high-level semantic information, i.e. target information, and promote the fusion of sub-features of each feature group.

[0102] Reference Figure 4 , step S103 is specifically: the spatial attention network includes a 3x3 convolution operation to further converge spatial information, then a mean-variance standardization operation is performed to generate a spatial attention weight map, and finally multiplied by the input element by element to obtain the final RGB_Atten and RGBX_Atten feature attention maps, as follows:

[0103] RGB_Atten = Norm(con3x3(RGB_HWAtten)) * RGB_HWAtten

[0104] RGBX_Atten = Norm(con3x3(RGBX_HWAtten)) * RGBX_HWAtten

[0105] Wherein Norm represents a mean standardization operation on the attention map after convolution.

[0106] The aforementioned S2-1 constructs a dynamic feature redefinition network, including the following sub-steps:

[0107] S2-1.1, the dynamic feature redefinition network comprises a 3X3 convolution network for reducing the channel dimension of the RGB_Atten and RGBX_Atten attention feature maps of the previous stage to 1, and then flattening as follows:

[0108] F_RGB_Atten = Flatten(Con3x3(RGB_Atten))

[0109] F_RGBX_Atten = Flatten(Con3x3(RGBX_Atten))

[0110] wherein Con3X3 represents a 3x3 convolution operation, Flatten represents a flattening operation, F_RGB_Atten∈R 1×HiWi represents a flattened feature sequence;

[0111] S2-1.2, by calculating the cosine similarity of the two flattened feature sequences to obtain a similarity score, multiplying the similarity score with the input modal feature to obtain the redefined feature maps R_RGB and R_RGBX as follows:

[0112] R_RGB = RGB_Atten*Cos_Simlar(F_RGB_Atten,F_RGBX_Atten)

[0113] R_RGBX = RGBX_Atten*Cos_Simlar(F_RGB_Atten,F_RGBX_Atten)

[0114] wherein Cos_Simlar represents calculating the cosine similarity score of two sequences.

[0115] Reference Figure 5 , the aforementioned step S2-2 constructs a cross-modal feature extraction network (FFM, Feature Fusion Module), specifically: input the redefined feature maps R_RGB and R_RGBX into the cross-modal feature extraction network, obtain the cross-modal global attention by calculating Cross self_attention, and obtain the final modal fusion feature maps Fuse_RGB and Fuse_RGBX as follows:

[0116] Fuse_RGB,Fuse_RGBX = Cross self_attention(R_RGB,R_RGBX)

[0117] wherein Cross self_attention represents a feature information operation of fusing R_RGB and R_RGBX.

[0118] The Cross self_attention in the cross-modal feature extraction network is an improvement based on the self_attention of the transformer. R_RGB and R_RGBX are respectively encoded and calculated to obtain three matrix vectors Q, K, and V containing the semantic representation of the input features. Then, the global attention maps Atten_RGB and Atten_RGBX are respectively calculated according to the self_attention. The global attention maps of the two modalities are multiplied and fused into Atten_fuse, as follows:

[0119]

[0120]

[0121] Atten_fuse = Atten_RGB * Atten_RGBX

[0122] The fused Atten_fuse is multiplied by the V of R_RGB, and then added to the V generated by the cross-modal multiplication of the global attention map Atten_RGB of R_RGB and R_RGBX, to realize the cross-fusion of modal information and obtain the Fuse_RGB of the fused RGBX information. As follows:

[0123] Fuse_RGB = Atten_fuse * V RGB + Atten_RGB * V RGBX

[0124] Fuse_RGBX = Atten_fuse * V RGBX + Atten_RGBX * V RGB

[0125] Since the inter-modal contains noise information of interference prediction. To this end, the application calculates the similarity between the modalities before feature fusion, obtains the similarity score by evaluating the correlation between the modal feature information, and adaptively weights the features of the calibrated modalities. The two modal features from the encoder are respectively generated into spatial attention maps through convolution group operations. Then, each subspace attention map is flattened to calculate their attention similarity. Due to the high sensitivity of the cosine distance to the direction and spatial factors, the application adopts the cosine distance to calculate the similarity score. The similarity score ranges from 0 to 1. The higher the value of the similarity score, the stronger the consistency of the different modal features. Finally, the application multiplies the similarity score with the input modal features to optimize the fusion ratio of the modal information. The Cross-self attention of the application is based on the traditional self-attention to enhance the feature extraction capability and modal information utilization across modalities. First, the two calibrated modal features are respectively encoded into Query (Q), Key (K), and Value (V). Through matrix multiplication, QK T The global attention map is calculated, and then the attention maps of the two modalities are multiplied as the common weight coefficient between the two modalities. At the same time, the respective global attention maps are multiplied with the V of the other modality through residual connection and added to the V of itself multiplied by the common weight coefficient. This operation realizes the information interaction across modalities and fully utilizes the feature information between different modalities.

[0126] The global attention map is calculated, and then the attention maps of the two modalities are multiplied as the common weight coefficient between the two modalities. At the same time, the respective global attention maps are multiplied with the V of the other modality through residual connection and added to the V of itself multiplied by the common weight coefficient. This operation realizes the information interaction across modalities and fully utilizes the feature information between different modalities.

[0127] Another aspect of the application provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method of any one of the application.

[0128] The application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the steps of the method of any one of the application.

[0129] Although the application has been described as above with reference to the preferred embodiments, it is not intended to limit the application. Those skilled in the art can make various modifications and improvements without departing from the spirit and scope of the application. Therefore, the protection scope of the application shall be subject to the scope defined by the claims.

Claims

1. A method based on multi-modal feature fusion of vehicle-mounted camera, characterized in that, Comprising the following steps: S1, feature attention stage: a grouped feature cross attention mechanism network is constructed, an RGBX picture generated by an RGB and a vehicle-mounted camera is input into a double-branch feature extraction network to obtain output feature maps F_RGB and F_RGBX of the i-th stage, and then the feature maps F_RGB and F_RGBX are input into the grouped feature cross attention mechanism network, spatial information of the two feature maps is cross-positioned, key features are extracted, and feature attention maps RGB_Atten and RGBX_Atten are obtained; specifically comprising the following sub-steps S101 to S103: S101, reshape operation is used to change F_RGB, F_RGBX ∈ R Ci×Hi×Wi to R G×Hi×Wi ; Wherein, Ci, Hi, Wi are the channel number, the length of the feature map and the width of the feature map of the output feature map of the i-th stage respectively, and G is the number of groups of the feature map; S102, the grouped feature map is input into an HW feature compression attention network to obtain a spatial attention code containing H and W two-dimensional features as follows: RGB_HWAtten=Se_HW(F_RGB), RGBX_HWAtten=Se_HW(F_RGBX); Wherein, RGB_HWAtten is the attention code generated by F_RGB through the HW feature compression attention network, and Se_HW function represents the operation of compressing and extracting H and W two-dimensional features; S103, then the attention code containing spatial coordinate information is aggregated through the channel information of the spatial attention network feature map to obtain the feature attention maps RGB_Atten and RGBX_Atten as follows: RGB_Atten=Spatial(RGB_HWAtten) RGBX_Atten=Spatial(RGBX_HWAtten) wherein the Spatial function represents a further extraction operation on the spatial attention of the attention encoding Se_Atten, and finally restores the grouped RGB_Atten∈R Ci×Hi×Wi ; S2, feature fusion stage: comprising: S2-1, a dynamic feature redefinition network is constructed, the feature attention maps RGB_Atten and RGBX_Atten are input into the dynamic feature redefinition network, the similarity score is obtained through the correlation of the feature attention maps RGB_Atten and RGBX_Atten, the corresponding features between modalities are adaptively weighted, and the redefined feature maps R_RGB and R_RGBX are obtained; specifically comprising the following sub-steps S2-1.1 to S2-1.2: S2-1.1, the dynamic feature redefinition network comprises a 3X3 convolution network, which is used to reduce the channel dimension of the RGB_Atten and RGBX_Atten attention feature maps of the previous stage to 1, and then flatten as follows: F_RGB_Atten=Flatten(Con3x3(RGB_Atten)) F_RGBX_Atten=Flatten(Con3x3(RGBX_Atten)) wherein, Con3X3 represents a 3x3 convolution operation, Flatten represents a flattening operation, F RGB Atten e R 1×HiWi represents the flattened feature sequence; S2-1.2, obtain a similarity score by calculating the cosine similarity of the two flattened feature sequences, multiply the similarity score with the input modal feature, and obtain the redefined feature maps R_RGB and R_RGBX, as follows: R_RGB = RGB_Atten * Cos_Simlar (F_RGB_Atten, F_RGBX_Atten) R_RGBX = RGBX_Atten * Cos_Simlar (F_RGB_Atten, F_RGBX_Atten) Wherein, Cos_Simlar represents calculating the cosine similarity score of two sequences; Step S2-2, constructing a cross-modal feature extraction network, inputting the redefined feature maps R_RGB and R_RGBX into the cross-modal feature extraction network, fusing the feature information of another modal, and obtaining the final modal fusion feature maps Fuse_RGB and Fuse_RGBX for input into the feature extraction network of the i+1 stage of the double branch to further extract features, specifically: inputting the redefined feature maps R_RGB and R_RGBX into the cross-modal feature extraction network, obtaining cross-modal global attention by calculating Cross self_attention, and obtaining the final modal fusion feature maps Fuse_RGB and Fuse_RGBX as follows: Fuse_RGB, Fuse_RGBX = Cross self_attention (R_RGB, R_RGBX) Wherein, Cross self_attention represents the feature information operation of fusing R_RGB and R_RGBX. 2.The method based on multi-modal feature fusion of vehicle-mounted camera according to claim 1, characterized in that, Step S102 includes the following sub-steps: S102-1, aggregate the feature maps RGBh_Atten and RGBXh_Atten containing W spatial coordinate information along the H direction by one-dimensional pooling operation, as follows: Wherein, c represents one channel of the input feature map, and W spatial information is compressed and encoded to obtain RGBh_Atten; h represents the height of a certain channel of the input feature map, w represents the width of a certain channel of the input feature map, and j represents the variable traversing the w pixels; S102-2, using the encoding-decoding structure of 3x3 hole convolution, after h_swish activation function, multiply with the input to obtain RGB_HAtten containing W spatial information, as follows: RGB_HAtten = h_swish (con3x3 (RGBh_Atten)) * F_RGB RGBX_HAtten = h_swish (con3x3 (RGBXh_Atten)) * F_RGBX S102-3, aggregate the feature maps RGBw_Atten and RGBXw_Atten containing H spatial coordinate information by one-dimensional pooling operation along the W direction through RGB_HAtten and RGBX_HAtten containing W spatial information, as follows: Wherein, w represents the width of one channel of the input feature map, and j represents the variable traversing the h pixels; S102-4, input the encoding-decoding structure of 3x3 hollow convolution, multiply the input after h_swish activation function to obtain RGB_HWAtten containing HW spatial information, as follows: RGB_HWAtten = h_swish(con3x3(RGBw_Atten))*RGB_HAtten RGBX_HWAtten = h_swish(con3x3(RGBXw_Atten))*RGBX_HAtten. 3.The method of claim 2, wherein, Step S103 is specifically: the spatial attention network includes a 3x3 convolution operation to further converge spatial information, then a mean-variance normalization operation is performed to generate a spatial attention weight map, and finally an element-wise multiplication is performed with the input to obtain the final RGB_Atten and RGBX_Atten feature attention map, as follows: RGB_Atten = Norm(Con3x3(RGB_HWAtten))*RGB_HWAtten RGBX_Atten = Norm(Con3x3(RGBX_HWAtten))*RGBX_HWAtten Where Norm represents the mean normalization operation on the attention map after convolution.

4. The multi-modal feature fusion method based on vehicle-mounted cameras according to claim 3, characterized in that, In the cross-modal feature extraction network, the Cross self_attention is an improved self_attention based on transformer. R_RGB and R_RGBX are encoded and calculated to obtain corresponding three matrix vectors Q, K and V containing input feature semantic representation. Then, the global attention map Atten_RGB and Atten_RGBX are calculated according to self_attention, and the global attention maps of the two modalities are multiplied and fused into Atten_fuse, as follows: Atten_fuse = Atten_RGB*Atten_RGBX The fused Atten_fuse is multiplied by the V of R_RGB, and then the V generated by the global attention map Atten_RGB of R_RGB cross-modally multiplied by R_RGBX is added to realize cross-fusion of modal information, to obtain Fuse_RGBX containing fused RGBX information and Fuse_RGB containing fused RGB information, as follows: Fuse_RGB = Atten_fuse * V RGB + Atten_RGB * V RGBX Fuse_RGBX = Atten_fuse * V RGBX + Atten_RGBX * V RGB .

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the method of any one of claims 1 to 4.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 4.

Images