Multimodal image fusion decision method, device, equipment and storage medium

Abstract

Embodiments of the present application provide a multi-modal image fusion decision method and device, equipment and a storage medium, relating to the technical field of image processing. The RGB encoding feature and the DVS encoding feature are obtained according to the RGB image and the DVS image corresponding to the current frame, the space-time mask matrix corresponding to the current frame is obtained, the RGB encoding feature, the DVS encoding feature and the space-time mask matrix are input into the attention fusion structure of a preset number of layers to be cascaded with each other for feature fusion to obtain the RGB fusion feature and the DVS fusion feature, the interaction feature obtained through information interaction based on the RGB fusion feature and the DVS fusion feature is obtained, and the decoder is used to obtain the prediction result of the current frame according to the RGB fusion feature, the DVS fusion feature and the interaction feature. In view of the noise generated by the influence of insufficient light on the RGB image in a dark scene and the interference caused by the deviation of dynamic information capture of the DVS image, the space-time mask matrix is introduced for suppression, the influence of the noise is reduced, and the scene dynamic information is supplemented through the interaction feature, so that the prediction accuracy in the dark scene is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a multi-modal image fusion decision method and device, equipment and a storage medium. BACKGROUND

[0002] In image prediction, especially in dark scenes, the RGB camera usually has problems such as reduced signal-to-noise ratio and lost texture information due to insufficient light, resulting in a significant decline in prediction performance. The event camera (Dynamic Vision Sensor, DVS) triggers an asynchronous event stream based on pixel-level brightness changes, has high dynamic range (more than 140 dB), low delay (microsecond-level response), and low power consumption, and can effectively capture the faint details of fast-moving targets. However, the event camera lacks continuous texture information in static scenes, and the sparsity of event stream data may lead to insufficient semantic understanding.

[0003] In related technologies, the rich semantic features of the RGB camera and the high dynamic response capability of the event camera are fused to improve the prediction performance of perception tasks, and are widely used in the fields of autonomous driving, security monitoring, etc. However, in extremely dark environments, noise suppression is not considered in the fusion process, resulting in a high error in the prediction result after fusion. SUMMARY

[0004] The main purpose of the embodiments of the present application is to propose a multi-modal image fusion decision method, device, equipment and storage medium to improve the prediction accuracy based on multi-modal image fusion in dark scenes.

[0005] To achieve the above purpose, the first aspect of the embodiments of the present application proposes a multi-modal image fusion decision method, comprising:

[0006] inputting the obtained RGB image and DVS image corresponding to the current frame into corresponding feature encoders for encoding to obtain RGB encoded features and DVS encoded features;

[0007] obtaining a space-time mask matrix corresponding to the current frame, inputting the RGB encoded features, the DVS encoded features and the space-time mask matrix into an attention fusion structure with a preset number of layers to perform feature fusion, to obtain RGB fusion features and DVS fusion features, and obtaining an interaction feature obtained by information interaction based on the RGB fusion features and the DVS fusion features;

[0008] inputting the RGB fusion features, the DVS fusion features and the interaction feature into a decoder for decoding to obtain a prediction result corresponding to the current frame.

[0009] In some embodiments, the inputting the RGB encoded feature, the DVS encoded feature and the spatio-temporal mask matrix into the preset number of attention fusion structures which are cascaded with each other for feature fusion to obtain RGB fusion features and DVS fusion features comprises:

[0010] For each of the attention fusion structures, a first input data and a second input data are obtained, an initial value of the first input data being the RGB encoded feature and an initial value of the second input data being the DVS encoded feature;

[0011] In the corresponding attention fusion structure, a first query vector is obtained according to the first input data, a first key vector and a first value vector are obtained according to the DVS encoded feature, a second key vector and a second value vector are obtained according to the second input data, a second query vector is obtained according to the DVS encoded feature, a first output data is calculated according to the spatio-temporal mask matrix, the first query vector, the first key vector and the first value vector, and a second output data is calculated according to the spatio-temporal mask matrix, the second query vector, the second key vector and the second value vector;

[0012] For the two cascaded attention fusion structures, the first output data of the previous one is the first input data of the latter, the second output data of the previous one is the second input data of the latter, the first output data of the last attention fusion structure is the RGB fusion feature, and the second output data is the DVS fusion feature.

[0013] In some embodiments, the RGB image is divided into a plurality of image blocks according to preset block information, and the calculation of the first output data according to the spatio-temporal mask matrix, the first query vector, the first key vector and the first value vector comprises:

[0014] A first attention matrix corresponding to each of the image blocks is calculated based on the first query vector and the first key vector;

[0015] A block mask matrix corresponding to the image block is obtained from the spatio-temporal mask matrix, and a product of the block mask matrix and the first attention matrix is calculated to obtain a first attention mask matrix corresponding to each of the image blocks;

[0016] Block output data corresponding to the image block is calculated based on the first attention matrix and the first value vector, and the first output data is obtained based on the block output data.

[0017] In some embodiments, before the obtaining of the block mask matrix corresponding to the image block in the current frame, the method further comprises:

[0018] generating an initial mask matrix corresponding to each block position according to the block information of the RGB image;

[0019] For each block position, at least one position circle is determined in a distance centering on the block position, and an initial mask weight value corresponding to each block position is determined in turn based on the order from near to far of the position circle, and the initial mask weight values of the block positions in the same position circle are the same;

[0020] The initial mask weight value is filled in the corresponding position of the initial mask matrix to obtain the block mask matrix of the current frame.

[0021] In some embodiments, before the initial mask weight value is filled in the corresponding position of the initial mask matrix to obtain the block mask matrix of the current frame, the method further comprises:

[0022] obtaining the initial mask weight value corresponding to each block position in the previous frame corresponding to the current frame as a reference weight value, and when the current frame is the first frame, the reference weight value is an initial value;

[0023] For each block position, updating the initial mask weight value in the current frame greater than the reference weight value.

[0024] In some embodiments, the information interaction based on the RGB fusion feature and the DVS fusion feature obtains an interaction feature, comprising:

[0025] The first output data and the second output data corresponding to each attention fusion structure are input into the corresponding feature memory structure for data processing;

[0026] In the feature memory structure, an interaction key vector and an interaction value vector are obtained according to the accumulation results of the first output data and the second output data, an initial interaction vector corresponding to the current attention fusion structure is obtained as an interaction query vector, and an initial value of the initial interaction vector is a preset vector;

[0027] According to the interaction query vector, the interaction key vector and the interaction value vector, cross attention vectors are obtained by attention calculation, and the interaction feature is obtained according to the cross attention vectors.

[0028] In some embodiments, the interaction feature is obtained according to the cross attention vectors, comprising:

[0029] The cross-attention vector is input into a multi-head attention structure for feature processing to obtain interaction output data, and the interaction output data corresponding to the last attention fusion structure is the interaction feature.

[0030] The cross-attention vector and the initial interaction vector are added to update the initial interaction vector.

[0031] To achieve the above object, a second aspect of the embodiment of the present application provides a multi-modal image fusion decision device, comprising:

[0032] The encoding module is configured to input the acquired RGB image and DVS image corresponding to the current frame into corresponding feature encoders for encoding to obtain RGB encoded features and DVS encoded features.

[0033] The fusion module is configured to acquire a space-time mask matrix corresponding to the current frame, input the RGB encoded features, the DVS encoded features and the space-time mask matrix into an attention fusion structure with a preset number of layers and cascaded with each other for feature fusion to obtain RGB fusion features and DVS fusion features, and acquire interaction features obtained through information interaction based on the RGB fusion features and the DVS fusion features.

[0034] The decoding module is configured to input the RGB fusion features, the DVS fusion features and the interaction features into a decoder for decoding to obtain a prediction result corresponding to the current frame.

[0035] To achieve the above object, a third aspect of the embodiment of the present application provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0036] To achieve the above object, a fourth aspect of the embodiment of the present application provides a storage medium, which is a storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0037] The multimodal image fusion decision method, apparatus, device, and storage medium proposed in this application involve inputting the acquired RGB image and DVS image corresponding to the current frame into corresponding feature encoders for encoding to obtain RGB encoded features and DVS encoded features. A spatiotemporal mask matrix corresponding to the current frame is then obtained. The RGB encoded features, DVS encoded features, and spatiotemporal mask matrix are input into a pre-defined cascaded attention fusion structure for feature fusion, resulting in RGB fusion features and DVS fusion features. Interaction features obtained through information interaction based on the RGB fusion features and DVS fusion features are also acquired. Finally, the RGB fusion features, DVS fusion features, and interaction features are input into a decoder for decoding to obtain the prediction result corresponding to the current frame. The fusion architecture in this application is a lightweight architecture, employing a pre-defined cascaded attention fusion structure to process RGB encoded features and DVS encoded features. The attention mechanism focuses on key information, avoiding the ineffective processing of redundant features. The cascaded structure, through a layered and progressive fusion method, ensures sufficient feature interaction while controlling the overall parameter scale and computational complexity of the model. In addition, a spatiotemporal mask matrix is ​​introduced to suppress noise in RGB images susceptible to insufficient illumination in dark scenes and interference in DVS images that may arise from deviations in capturing dynamic information, thereby reducing the impact of noise on related fusion features. Simultaneously, interactive features are used to supplement scene dynamic information, comprehensively improving prediction accuracy in dark scenes. Attached Figure Description

[0038] Figure 1 This is a flowchart of the multimodal image fusion decision method provided in the embodiments of this application.

[0039] Figure 2 This is an overall schematic diagram of the multimodal image fusion decision method provided in the embodiments of this application.

[0040] Figure 3 This is a flowchart provided in this application embodiment, which shows how to perform feature fusion by inputting RGB encoded features, DVS encoded features and spatiotemporal mask matrix into an attention fusion structure that is cascaded with each other at a preset number of layers, to obtain RGB fused features and DVS fused features.

[0041] Figure 4 This is a schematic diagram of the cascaded attention fusion structure provided in the embodiments of this application.

[0042] Figure 5 This is a flowchart illustrating the generation process of the spatiotemporal mask matrix provided in an embodiment of this application.

[0043] Figure 6 This is a schematic diagram of the block mask matrix provided in the embodiments of this application.

[0044] Figure 7is a flowchart of updating the initial value of the mask weight by the embodiment of the present application.

[0045] Figure 8 is a flowchart of calculating the first output data according to the space-time mask matrix, the first query vector, the first key vector and the first value vector provided by the embodiment of the present application.

[0046] Figure 9 is a flowchart of obtaining the interaction feature based on the RGB fusion feature and the DVS fusion feature provided by the embodiment of the present application.

[0047] Figure 10 is an execution process schematic diagram of the feature memory structure provided by the embodiment of the present application.

[0048] Figure 11 is a flowchart of obtaining the interaction feature according to the cross-attention vector provided by the embodiment of the present application.

[0049] Figure 12 is a structure block diagram of the multi-modal image fusion decision device provided by another embodiment of the present application.

[0050] Figure 13 is a hardware structure schematic diagram of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0052] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device or the order in the flowchart.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0054] Firstly, the several terms involved in the present application are analyzed:

[0055] Artificial Intelligence (AI): is a new technical science that studies, develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; Artificial intelligence is a branch of computer science, artificial intelligence tries to understand the essence of intelligence, and produce a new intelligent machine that can react in a similar way to human intelligence, including robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain optimal results.

[0056] In image prediction, especially in dark scenes, RGB cameras often have problems such as reduced signal-to-noise ratio, loss of texture information, and significant decline in prediction performance due to insufficient light. The event camera (Dynamic Vision Sensor, DVS) triggers an asynchronous event stream based on pixel-level brightness changes, has high dynamic range (more than 140dB), low latency (microsecond-level response), and low power consumption, and can effectively capture the faint details of fast-moving targets. However, event cameras lack continuous texture information in static scenes, and the sparsity of event stream data may lead to insufficient semantic understanding.

[0057] In related technologies, the rich semantic features of RGB cameras and the high dynamic response capability of event cameras are fused to improve the prediction performance of perception tasks and are widely used in automatic driving, security monitoring and other fields. Generally, there are two types of fusion methods, one is single-frame fusion, which only considers the fusion of RGB and DVS image frames at the current time, and the other is multi-frame fusion, which only considers the fusion of RGB and DVS image frames at multiple times or multiple perspectives. Both of these two types include static fusion or multi-scale fusion. For example, ResNet is used to train DVS and RGB images to predict the turning angle, which is a static fusion method that does not consider dynamic information of multiple frames and directly fuses multiple scale features. However, the single-frame method obviously lacks more information to supplement feature fusion, which leads to poor prediction results in dark scenes if the current frame has a lot of noise. The multi-frame method does not consider noise suppression, which leads to high errors in the prediction results after fusion. In addition, most multi-scale fusion methods based on ResNet also consume a lot of training memory, and the fusion architecture is not light enough.

[0058] Based on this, the embodiment of the application provides a multi-modal image fusion decision method, device, equipment and storage medium, a light-weight fusion architecture is set, an attention fusion structure with a preset number of layers is cascaded to process RGB coded features and DVS coded features, key information is focused by using the attention mechanism, invalid processing of redundant features is avoided, and the cascaded structure controls the overall parameter size and the calculation complexity of the model by using a hierarchical progressive fusion mode while ensuring sufficient feature interaction. In addition, a space-time mask matrix is introduced, noise generated by the influence of insufficient light on the RGB image in a dark scene and interference that may occur due to the deviation of dynamic information capture of the DVS image are suppressed, and the influence of noise on related fusion features is reduced. At the same time, the scene dynamic information is supplemented by interactive features, and the prediction accuracy in a dark scene is comprehensively improved.

[0059] The embodiment of the application provides a multi-modal image fusion decision method, device, equipment and storage medium, and the method is specifically described through the following embodiment. First, the multi-modal image fusion decision method in the embodiment of the application is described.

[0060] The embodiment of the application can acquire and process related data based on artificial intelligence technology. The artificial intelligence (AI) is a theory, method, technology and application system for using a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0061] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware and software technologies. Artificial intelligence basic technologies generally include sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning and other fields.

[0062] The multi-modal image fusion decision method provided by the embodiments of the present application relates to the technical field of image processing. The multi-modal image fusion decision method provided by the embodiments of the present application can be applied to a terminal, can also be applied to a server, and can also be a computer program running in the terminal or the server. For example, the computer program can be a native program or a software module in an operating system; can be a native application (APP), that is, a program that needs to be installed in an operating system to run, such as a client supporting multi-modal image fusion decision, that is, a program that can run only by being downloaded into a browser environment; and can also be an applet that can be embedded into any APP. In summary, the above computer program can be any form of application program, module or plug-in. The terminal communicates with the server through a network. The multi-modal image fusion decision method can be executed by the terminal or the server, or cooperatively executed by the terminal and the server.

[0063] In some embodiments, the terminal can be a smartphone, a tablet computer, a notebook computer, a desktop computer or a smart watch, etc. In addition, the terminal can also be a smart vehicle-mounted device. The smart vehicle-mounted device applies the multi-modal image fusion decision method of the embodiments to provide related services and improve the driving experience. The server can be a stand-alone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms; or a service node in a blockchain system, the service nodes in the blockchain system form a peer-to-peer (P2P) network, and the P2P protocol is an application layer protocol running on the transmission control protocol (TCP) protocol. The terminal and the server can be connected through a communication connection mode such as Bluetooth, universal serial bus (USB) or network, and the embodiments do not limit this.

[0064] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0065] The multimodal image fusion decision method in the embodiments of this application is described below.

[0066] Figure 1 This is an optional flowchart of the multimodal image fusion decision method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps 110 to 130. It is also understood that this embodiment... Figure 1 The order of steps 110 to 130 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0067] Step 110: Input the RGB image and DVS image corresponding to the current frame into the corresponding feature encoder for encoding to obtain RGB encoded features and DVS encoded features.

[0068] In one embodiment, reference is made to Figure 2 , Figure 2 This is an overall schematic diagram of the multimodal image fusion decision method provided in this application embodiment. First, multiple consecutive frames of RGB images and multiple frames of DVS images are acquired based on time series. Then, the time frames of the RGB images and DVS images are aligned using time-series alignment. The number of time frames can be 2, 4, 8, etc., set according to actual needs. The RGB images and DVS images are obtained from images captured in the same dark scene. Here, "dark" does not mean completely black, but only indicates poor lighting conditions in the scene, resulting in images containing significant noise that cannot be directly used in the image recognition process.

[0069] Then, the RGB image and DVS image corresponding to each time frame are processed. (Refer to...) Figure 2First, the RGB image and the DVS image are fed into their respective feature encoders for encoding, resulting in the RGB encoded feature HR corresponding to the RGB image and the DVS encoded feature HD corresponding to the DVS image.

[0070] In one embodiment, Figure 2 Both feature encoders are multi-head self-attention (MHSA) structures. The RGB and DVS images are divided into multiple image blocks using the same block information; for example, the RGB image size is... At this time, it is divided into The image is divided into blocks, and the size of each block is [size missing]. In the feature encoder, for each input image block, the data of the image block is first flattened into a sequence, which is then mapped to the model dimension while adding positional encoding to preserve spatial information. The positionally encoded data is then projected using three sets of learnable parameters to generate relevant query vectors, key vectors, and value vectors, and split according to the number of attention heads. Next, for each head, the attention score between image blocks is calculated, thus obtaining the attention weight. The corresponding value vectors are then weighted and summed using the attention weight to obtain the output for each attention head. All outputs are concatenated to obtain the corresponding encoded features. It can be understood that both RGB encoded features and DVS encoded features contain block-level encoded features for each corresponding image block.

[0071] Step 120: Obtain the spatiotemporal mask matrix corresponding to the current frame. Input the RGB encoded features, DVS encoded features and spatiotemporal mask matrix into an attention fusion structure with a preset number of cascaded layers for feature fusion to obtain RGB fusion features and DVS fusion features. Then, obtain the interaction features obtained by information interaction based on the RGB fusion features and DVS fusion features.

[0072] In one embodiment, feature processing is then performed based on RGB encoding features and DVS encoding features. (Refer to...) Figure 3 , Figure 3 This application provides a flowchart illustrating the process of fusing RGB coded features, DVS coded features, and a spatiotemporal mask matrix into a pre-defined number of cascaded attention fusion structures to obtain RGB fused features and DVS fused features. The flowchart specifically includes the following steps:

[0073] Step 310: For each attention fusion structure, obtain the first input data and the second input data.

[0074] In one embodiment, reference is made to Figure 4 , Figure 4is a schematic diagram of the attention fusion structure cascaded with each other provided by an embodiment of the present application. Figure 4 Each attention fusion structure is boxed with a dashed line, wherein each attention fusion structure comprises a first fusion layer and a second fusion layer. One input of the first fusion layer in each attention fusion structure is referred to as first input data, and one input of the second fusion layer is referred to as second input data.

[0075] For the first attention fusion structure, the first input data is RGB encoded features, and the second input data is DVS encoded features. That is, the initial value of the first input data is RGB encoded features, and the initial value of the second input data is DVS encoded features.

[0076] Step 320: In the corresponding attention fusion structure, a first query vector is obtained according to the first input data, a first key vector and a first value vector are obtained according to the DVS encoded features, a second key vector and a second value vector are obtained according to the second input data, a second query vector is obtained according to the DVS encoded features, a first output data is calculated according to the spatiotemporal mask matrix, the first query vector, the first key vector and the first value vector, and a second output data is calculated according to the spatiotemporal mask matrix, the second query vector, the second key vector and the second value vector.

[0077] In an embodiment, taking any one of the attention fusion structures as an example, the input data of the first fusion layer comprises the first input data and the DVS encoded features, and the input data of the second fusion layer comprises the second input data and the RGB encoded features.

[0078] At this time, in the first fusion layer, a first query vector is obtained according to the first input data, that is, the first input data and the learnable first query projection weight W Q1 are multiplied to obtain the first query vector Q1, the DVS encoded features and the first key projection weight W K1 are multiplied to obtain the first key vector K1, and the DVS encoded features and the first value projection weight W V1 are multiplied to obtain the first value vector V1. Similarly, in the second fusion layer, the second input data and the learnable second key projection weight W K2 are multiplied to obtain the second key vector, and the second input data and the learnable second value projection weight W V2 are multiplied to obtain the second value vector, and a second query vector is obtained according to the DVS encoded features. It can be understood that there is a corresponding set of query vectors, value vectors and key vectors for each image block.

[0079] Next, the first output data is calculated according to the spatiotemporal mask matrix, the first query vector, the first key vector and the first value vector in the first fusion layer, and the second output data is calculated according to the spatiotemporal mask matrix, the second query vector, the second key vector and the second value vector in the second fusion layer. The calculation process of the first output data is described below as an example, and the calculation process of the second output data is similar to that of the first output data.

[0080] The acquisition process of the spatiotemporal mask matrix is described here. Embodiments of the present application use the spatiotemporal mask matrix to reduce the influence of noise of the RGB and DVS sensors in a dark scene on the fusion feature prediction accuracy. In an embodiment, with reference to Figure 5 , Figure 5 is a generation flowchart of the spatiotemporal mask matrix provided by embodiments of the present application, specifically comprising the following steps:

[0081] Step 510: generating an initial mask matrix corresponding to each block position according to the block information of the RGB image.

[0082] In an embodiment, since the RGB image and the DVS image are both segmented into multiple image blocks based on the same block information, for example, the image blocks in the above example. At this time, an initial mask matrix is generated for the block position of each image block, and the size of the initial mask matrix is .

[0083] Step 520: for each block position, at least one position ring is determined according to the distance, and based on the order from near to far of the position ring, the initial mask weight value corresponding to each block position is determined in turn.

[0084] In an embodiment, the mask matrix corresponding to each block position is called a block mask matrix. With reference to Figure 6 , Figure 6 is a schematic diagram of the block mask matrix provided by embodiments of the present application. Figure 6 The black square in the center of the black square represents the selected block position, and it can be seen that the black square is divided into three complete position rings, i.e. q1, q2 and q3 shown in the figure, and half a position ring, i.e. q4 shown in the figure. At this time, it can be considered that the distance between the center position in the smallest position ring and the center position is the closest, and the distance between the center position in the half position ring and the center position is the farthest.

[0085] Therefore, in order from near to far in the position circle, the mask weight initial value corresponding to each sub-block position is determined in turn, and spatial mask is realized. At this time, the mask weight initial values of the sub-block positions in the same position circle are the same, and the mask weight initial value is smaller as the distance is farther. That is, the mask weight initial value of the position circle corresponding to q1 is greater than the mask weight initial value of the position circle corresponding to q2, the mask weight initial value of the position circle corresponding to q2 is greater than the mask weight initial value of the position circle corresponding to q3, and the mask weight initial value of the position circle corresponding to q3 is greater than the mask weight initial value of the position circle corresponding to q4. In this way, the place where each image sub-block is located has the maximum weight, such as 1.0-2.0, and the weights of the remaining image sub-blocks gradually decrease to close to 0 as the distance in space increases. In this way, the RGB coding feature can greatly reduce the useless noise of the DVS image in the learning process of attention in the far space distance.

[0086] Step 530: Fill the mask weight initial value in the initial mask matrix at the corresponding position to obtain the sub-block mask matrix of the current frame.

[0087] In an embodiment, the mask weight initial value at this time can realize spatial mask, but the embodiment of the present application considers that there is noise accumulation phenomenon in the DVS image of multiple frames in the time dimension, and there may be historical noise, so the mask weight initial value is also updated to meet the time mask effect.

[0088] In an embodiment, referring to Figure 7 , Figure 7 is a flowchart for updating the mask weight initial value by the embodiment of the present application, and specifically includes the following steps:

[0089] Step 710: Obtain the mask weight initial value corresponding to each sub-block position in the previous frame corresponding to the current frame as a reference weight value.

[0090] In an embodiment, according to the position of the current frame in the time sequence, the mask weight initial value corresponding to each sub-block position in the previous frame is selected as the reference weight value. If the current frame is the first frame, the reference weight value is the mask weight initial value of the first frame.

[0091] Step 720: For each sub-block position, adjust the mask weight initial value in the current frame to be greater than the reference weight value, and update the mask weight initial value.

[0092] In an embodiment, for the mask weight initial values corresponding to each sub-block position in the previous two time frames, if the mask weight initial value of the current frame is greater than the reference weight value, the mask weight initial value of the current frame is reduced for updating. That is, for the same image sub-block, the mask weight initial value of the current frame is the largest as time goes on.

[0093] In an embodiment, the initial mask weight values of the last update are filled in the corresponding positions in the initial mask matrix to obtain a block mask matrix of the current frame, and then the block mask matrix of each block position is used to form a space-time mask matrix of the current frame.

[0094] It can be understood that, since the number of time frames to be processed can be known in advance, the corresponding block mask matrix can be generated in advance for each block position of each frame, which satisfies two conditions: the first is that the corresponding mask weight initial value becomes larger as the time frame progresses, and the second is that the mask weight initial value of the block position is the largest, and the mask weight initial value corresponding to the block position becomes smaller as the distance between other block positions and it becomes larger.

[0095] In an embodiment, referring to Figure 8 , Figure 8 is a flowchart provided by the embodiment of the application for calculating the first output data according to the space-time mask matrix, the first query vector, the first key vector and the first value vector, and specifically includes the following steps:

[0096] Step 810: calculating at least one first attention matrix corresponding to each image block based on the first query vector and the first key vector.

[0097] In an embodiment, assuming that the number of attention heads in the first fusion layer is h, the total number of image blocks is M, and taking the i-th image block and the j-th attention head as an example, first, in the j-th attention head, the component corresponding to the first query vector is At this time, the attention score between the i-th image block and all other image blocks is calculated, denoted as:

[0098]

[0099] wherein, denotes the dimension of the attention head.

[0100] Then, the attention score is normalized to obtain the corresponding attention weight, denoted as:

[0101]

[0102] At this time, all the attention weights are formed into a matrix to obtain the first attention matrix of the j-th attention head.

[0103] Step 820: obtaining the block mask matrix corresponding to the image block from the space-time mask matrix, calculating the product of the block mask matrix and the first attention matrix, and obtaining the first attention mask matrix corresponding to each image block.

[0104] In an embodiment, it is assumed that the patch mask matrix corresponding to the i-th image patch is Maski, and thus for the j-th attention head, the product of the patch mask matrix Maski and the first attention matrix is calculated, and the first attention mask matrix corresponding to the i-th image patch in the j-th attention head is obtained.

[0105] Step 830: calculating patch output data corresponding to the image patch based on the first attention mask matrix and the first value vector, and obtaining the first output data based on the patch output data.

[0106] In an embodiment, the first attention mask matrix corresponding to the j-th attention head is weighted and aggregated with the components of the first value vector corresponding to the j-th attention head to obtain the output data corresponding to the j-th attention head, and the output data corresponding to all attention heads is spliced to obtain the patch output data corresponding to the i-th image patch. All patch output data is summarized to obtain the first output data.

[0107] In an embodiment, the second output data corresponding to the second fusion layer is obtained in the same way. According to Figure 4 It can be known from the structural diagram of the structure that for the two cascaded attention fusion structures, the first output data of the previous attention fusion structure is the first input data of the subsequent attention fusion structure, the second output data of the previous attention fusion structure is the second input data of the subsequent attention fusion structure, and the first output data of the last attention fusion structure is the RGB fusion feature and the second output data is the DVS fusion feature.

[0108] It can be seen that the embodiment of the present application realizes more in-depth interactive fusion of features through the lightweight fusion architecture shown in Figure 4 The preset number of layers of the attention fusion structure can be set to 1, 2, 3, …, and it can be understood that as the number of layers increases, the calculation complexity increases, and thus it can be adjusted according to the specific task requirements. In combination with Figure 4The fusion process is divided into two branches by the first fusion layer and the second fusion layer. The upper branch can be regarded as an RGB branch, which obtains a corresponding query vector according to the RGB encoded feature HR, and obtains a corresponding key vector and a value vector according to the DVS encoded feature. The corresponding query vector is updated and transmitted between each layer of the attention fusion structure. The query vector is refined through the first fusion layer, so as to optimize the representation of the weight matrix related to the DVS feature. The lower branch can be referred to as a DVS branch, which obtains a corresponding key vector and a value vector according to the RGB encoded feature HR, and obtains a corresponding query vector according to the DVS encoded feature. The corresponding key vector and the value vector are updated and transmitted between each layer of the attention fusion structure. This is because the query vector usually represents the focus of the model to learn at present. Therefore, the RGB encoded feature is used as the value vector and the key vector to refine through the second fusion layer, so as to enhance the attention of the model to the DVS information, enrich the learnable RGB feature, and enhance the robustness of the model. It should be noted that, since the events generated by the DVS camera are sparse, the DVS information will not be refined in the two branches, so as to avoid losing more features.

[0109] In an embodiment, in combination with Figure 2 In the fusion architecture, a feature memory structure is further introduced for further feature extraction. Referring to Figure 9 , Figure 9 is a flowchart for obtaining an interaction feature based on an RGB fusion feature and a DVS fusion feature according to the embodiment of the application, and specifically includes the following steps:

[0110] Step 910: input the first output data and the second output data corresponding to each attention fusion structure into the corresponding feature memory structure for data processing.

[0111] In an embodiment, after the execution of each attention fusion structure is completed, the corresponding first output data and the second output data are obtained and input into the feature memory structure for data processing, and the feature memory structure is used to perform a data processing process.

[0112] Step 920: in the feature memory structure, obtain an interaction key vector and an interaction value vector according to the accumulation result of the first output data and the second output data, and obtain an initial interaction vector corresponding to the current attention fusion structure as an interaction query vector.

[0113] In an embodiment, referring to Figure 10 , Figure 10An execution process schematic diagram of the feature memory structure provided by the embodiment of the present application is provided. Taking the data processing process of the i th feature memory structure as an example. The first output data FRi and the second output data FDi corresponding to the i th attention fusion structure are obtained, the first output data and the second output data are accumulated to obtain an accumulation result, and then the accumulation result is multiplied by the corresponding key projection weight and value projection weight to obtain the corresponding interaction key vector K' and interaction value vector V'. Then the initial interaction vector G corresponding to the current attention fusion structure is obtained, which is taken as the interaction query vector Q', wherein the initial value of the initial interaction vector is a preset vector, and the preset vector can be a set of learnable parameters.

[0114] Step 930: performing attention calculation according to the interaction query vector, the interaction key vector and the interaction value vector to obtain a cross-attention vector, and obtaining the interaction feature according to the cross-attention vector.

[0115] In an embodiment, in combination with Figure 10 , the attention mechanism is used to perform attention calculation according to the interaction query vector, the interaction key vector and the interaction value vector to obtain a cross-attention vector, and then the interaction feature is obtained according to the cross-attention vector.

[0116] In an embodiment, referring to Figure 11 , Figure 11 is a flowchart for obtaining the interaction feature according to the cross-attention vector provided by the embodiment of the present application, and specifically includes the following steps:

[0117] Step 1110: inputting the cross-attention vector into a multi-head attention structure for feature processing to obtain interaction output data.

[0118] In an embodiment, the cross-attention vector also needs to be input into a multi-head attention structure for attention calculation to obtain interaction output data FG'. The interaction output data corresponding to the last attention fusion structure is taken as the interaction feature FG.

[0119] Step 1120: adding the cross-attention vector and the initial interaction vector to update the initial interaction vector.

[0120] In an embodiment, the initial interaction vector G is a preset vector when the first attention fusion structure is calculated, and in the subsequent execution calculation process, the initial interaction vector is updated with the cross-attention vector. Referring to Figure 10 , specifically, when updating, the cross-attention vector and the current initial interaction vector are added to complete the update of the initial interaction vector. That is, with the increase of the number of executions, the initial interaction vector will contain the relevant information of all previous execution processes.

[0121] Step 130: input the RGB fusion feature, the DVS fusion feature and the interaction feature into a decoder to obtain a prediction result corresponding to the current frame.

[0122] In an embodiment, the decoder can be composed of multiple Transformers. In decoding, first, the RGB fusion feature, the DVS fusion feature and the interaction feature are spliced to obtain a decoding input feature, and then the decoding input feature is converted to a dimension corresponding to the decoder. Then, at least the following processes are performed in the decoder: normalization, mean pooling, activation operation, projection, etc., to obtain a prediction result corresponding to the current frame. It can be understood that the prediction result is different in different application scenarios. For example, in the scenarios of automatic driving and unmanned aerial vehicle control, the prediction result can be decision action information. This embodiment does not limit the application scenario.

[0123] The multi-modal image fusion decision method in the embodiment of the application solves the problem of large parameter quantity in the multi-scale fusion method based on Resnet, and realizes a lighter attention fusion framework. Based on the new fusion framework, the influence of the noise of the RGB and DVS sensors in a dark scene on the prediction accuracy of the fusion feature is reduced by combining a space-time mask matrix, and at the same time, the past learned intermediate fusion feature information is stored by the feature memory mechanism, thereby improving the fusion prediction accuracy from the principle and technical level.

[0124] The technical scheme provided in the embodiment of the application encodes the obtained RGB image and DVS image corresponding to the current frame into RGB encoding features and DVS encoding features respectively by inputting the obtained RGB image and DVS image corresponding to the current frame into corresponding feature encoders, inputs the RGB encoding features, the DVS encoding features and a space-time mask matrix into a preset number of layers of attention fusion structures that are cascaded with each other to perform feature fusion, obtains RGB fusion features and DVS fusion features, and obtains interaction features obtained by performing information interaction based on the RGB fusion features and the DVS fusion features, and then inputs the RGB fusion features, the DVS fusion features and the interaction features into a decoder to obtain a prediction result corresponding to the current frame. The fusion architecture in the embodiment of the application is a lightweight architecture, the RGB encoding features and the DVS encoding features are processed by using the preset number of layers of attention fusion structures that are cascaded with each other, the key information is focused by using the attention mechanism, and the invalid processing of redundant features is avoided, and the cascaded structure controls the overall parameter size and the calculation complexity of the model by using the hierarchical and progressive fusion mode while ensuring the sufficiency of feature interaction. In addition, the space-time mask matrix is introduced to suppress the noise of the RGB image caused by insufficient light in a dark scene and the interference of the DVS image caused by the deviation of dynamic information capture, and to reduce the influence of the noise on the related fusion features. At the same time, the scene dynamic information is supplemented by the interaction features, and the prediction accuracy in a dark scene is comprehensively improved.

[0125] The embodiment of the present application further provides a multi-modal image fusion decision device, which can implement the multi-modal image fusion decision method. Figure 12 The device comprises:

[0126] The encoding module 1210 is configured to input the acquired RGB image and DVS image corresponding to the current frame into corresponding feature encoders respectively for encoding to obtain RGB encoded features and DVS encoded features.

[0127] The fusion module 1220 is configured to acquire a space-time mask matrix corresponding to the current frame, input the RGB encoded features, the DVS encoded features and the space-time mask matrix into an attention fusion structure with a preset number of layers which are cascaded with each other for feature fusion to obtain RGB fusion features and DVS fusion features, and acquire interactive features obtained through information interaction based on the RGB fusion features and the DVS fusion features.

[0128] The decoding module 1230 is configured to input the RGB fusion features, the DVS fusion features and the interactive features into a decoder for decoding to obtain a prediction result corresponding to the current frame.

[0129] The specific implementation of the multi-modal image fusion decision device of the embodiment is basically the same as the specific implementation of the multi-modal image fusion decision method, and will not be repeated here.

[0130] The embodiment of the present application further provides an electronic device, which comprises:

[0131] at least one memory;

[0132] at least one processor;

[0133] at least one program;

[0134] The program is stored in the memory, and the processor executes the at least one program to implement the multi-modal image fusion decision method of the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0135] Please refer to Figure 13 , Figure 13 which illustrates the hardware structure of the electronic device of another embodiment, and the electronic device comprises:

[0136] The processor 1301 can be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0137] The memory 1302 can be implemented by a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1302 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 1302 and are called and executed by the processor 1301 to implement the multi-modal image fusion decision method of the embodiments of the present application.

[0138] The input / output interface 1303 is configured to implement information input and output.

[0139] The communication interface 1304 is configured to implement the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0140] The bus 1305 is configured to transmit information between various components (for example, the processor 1301, the memory 1302, the input / output interface 1303, and the communication interface 1304) of the device.

[0141] The processor 1301, the memory 1302, the input / output interface 1303, and the communication interface 1304 are connected to each other through the bus 1305 to realize the communication connection between the device.

[0142] The embodiments of the present application also provide a storage medium. The storage medium is a storage medium, and the storage medium stores a computer program. The computer program is executed by the processor to implement the multi-modal image fusion decision method.

[0143] The memory, as a non-transitory storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] The multi-modal image fusion decision method, device, equipment and storage medium provided by the embodiments of the present application are characterized in that the RGB image and the DVS image corresponding to the current frame obtained are respectively input into corresponding feature encoders for encoding to obtain RGB encoded features and DVS encoded features, a spatiotemporal mask matrix corresponding to the current frame is obtained, the RGB encoded features, the DVS encoded features and the spatiotemporal mask matrix are input into a preset number of layers of attention fusion structures that are cascaded with each other for feature fusion to obtain RGB fusion features and DVS fusion features, and an interaction feature obtained based on information interaction of the RGB fusion features and the DVS fusion features is obtained, the RGB fusion features, the DVS fusion features and the interaction feature are input into a decoder for decoding to obtain a prediction result corresponding to the current frame. In the embodiments of the present application, the fusion architecture is a lightweight architecture, the RGB encoded features and the DVS encoded features are processed by using the preset number of layers of attention fusion structures that are cascaded with each other, the key information is focused by using the attention mechanism, and invalid processing of redundant features is avoided, and the cascaded structure controls the overall parameter size and the calculation complexity of the model by using a hierarchical progressive fusion mode while ensuring sufficient feature interaction. In addition, the spatiotemporal mask matrix is introduced to suppress the noise of the RGB image caused by insufficient light in a dark scene and the interference of the DVS image caused by the deviation of dynamic information capture, and to reduce the influence of the noise on the related fusion features. At the same time, the interaction feature supplements the scene dynamic information, and the prediction accuracy in the dark scene is comprehensively improved.

[0145] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0146] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than those shown in the figures, or combine certain steps or different steps.

[0147] The apparatus embodiments described above are merely exemplary, and units described as separate components may or may not be physically separate, i.e., may be located in one place, or may be distributed over multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments.

[0148] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof.

[0149] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims of the foregoing drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so construed can be interchanged, such that, for example, a case described herein as a first case can be implemented as a second case or a third case, and vice versa, as appropriate, to produce the embodiments of the present application described herein. In addition, the terms "comprising" and "including" and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or apparatus that includes a list of steps or units not necessarily limited to those clearly identified as such, but can include other not clearly recited steps or units inherent in such process, method, product or apparatus.

[0150] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c, can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0151] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0152] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0153] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0154] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the technical solutions or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, and various program storage media.

[0155] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A multi-modal image fusion decision method, characterized in that, The method comprises the following steps: input the acquired RGB image and DVS image corresponding to the current frame into corresponding feature encoders for encoding to obtain RGB encoded features and DVS encoded features; acquire a space-time mask matrix corresponding to the current frame, input the RGB encoded features, the DVS encoded features and the space-time mask matrix into a preset number of attention fusion structures which are cascaded with each other for feature fusion, for each attention fusion structure, acquire first input data and second input data, the initial value of the first input data is the RGB encoded features, and the initial value of the second input data is the DVS encoded features; in the corresponding attention fusion structure, a first query vector is obtained according to the first input data, a first key vector and a first value vector are obtained according to the DVS encoded features, a second key vector and a second value vector are obtained according to the second input data, a second query vector is obtained according to the DVS encoded features, a first output data is calculated according to the space-time mask matrix, the first query vector, the first key vector and the first value vector, and a second output data is calculated according to the space-time mask matrix, the second query vector, the second key vector and the second value vector; for two cascaded attention fusion structures, the first output data of the former is the first input data of the latter, the second output data of the former is the second input data of the latter, the first output data of the last attention fusion structure is the RGB fusion features, and the second output data is the DVS fusion features, and the first output data and the second output data of each attention fusion structure are input into a corresponding feature memory structure for data processing; in the feature memory structure, an interaction key vector and an interaction value vector are obtained according to the accumulation results of the first output data and the second output data, an initial interaction vector corresponding to the current attention fusion structure is acquired as an interaction query vector, and the initial value of the initial interaction vector is a preset vector; an interaction feature is obtained by performing attention calculation according to the interaction query vector, the interaction key vector and the interaction value vector; the RGB fusion features, the DVS fusion features and the interaction feature are input into a decoder for decoding to obtain a prediction result corresponding to the current frame.

2. The multi-modal image fusion decision method of claim 1, wherein, The RGB image is divided into a plurality of image blocks according to preset block information, and the first output data is calculated according to the space-time mask matrix, the first query vector, the first key vector and the first value vector, which comprises the following steps: calculate a first attention matrix corresponding to each image block based on the first query vector and the first key vector; acquire a block mask matrix corresponding to the image block from the space-time mask matrix, calculate the product of the block mask matrix and the first attention matrix to obtain a first attention mask matrix corresponding to each image block; The method further comprises:

3. The multi-modal image fusion decision method of claim 2, wherein, The method further comprises: generating an initial mask matrix corresponding to each block position according to the block information of the RGB image; determining an initial mask weight value corresponding to each block position in sequence based on the order from near to far of at least one position circle centered on the block position according to the distance; filling the initial mask weight value in the corresponding position of the initial mask matrix to obtain the block mask matrix of the current frame.

4. The multi-modal image fusion decision method of claim 3, wherein, The method further comprises: obtaining the initial mask weight value corresponding to each block position in the previous frame corresponding to the current frame as a reference weight value, and the reference weight value is an initial value when the current frame is a first frame; updating the initial mask weight value in the current frame greater than the reference weight value for each block position.

5. The multi-modal image fusion decision method of claim 1, wherein, The method further comprises: inputting the cross-attention vector into a multi-head attention structure for feature processing to obtain interaction output data, and the interaction output data corresponding to the last attention fusion structure is the interaction feature; adding the cross-attention vector and the initial interaction vector to update the initial interaction vector.

6. A multi-modal image fusion decision apparatus, characterized by, The method further comprises: an encoding module configured to input the obtained RGB image and DVS image corresponding to the current frame into corresponding feature encoders respectively to obtain RGB encoded features and DVS encoded features; The fusion module is configured to obtain a space-time mask matrix corresponding to the current frame, input the RGB encoded feature, the DVS encoded feature, and the space-time mask matrix into a preset number of attention fusion structures that are cascaded with each other to perform feature fusion, obtain first input data and second input data for each of the attention fusion structures, and set an initial value of the first input data as the RGB encoded feature and an initial value of the second input data as the DVS encoded feature; in the corresponding attention fusion structure, obtain a first query vector according to the first input data, obtain a first key vector and a first value vector according to the DVS encoded feature, obtain a second key vector and a second value vector according to the second input data, obtain a second query vector according to the DVS encoded feature, calculate first output data according to the space-time mask matrix, the first query vector, the first key vector, and the first value vector, and calculate second output data according to the space-time mask matrix, the second query vector, the second key vector, and the second value vector; for two cascaded attention fusion structures, the first output data of a previous one is the first input data of a next one, the second output data of the previous one is the second input data of the next one, the first output data of the last attention fusion structure is the RGB fusion feature, and the second output data of the last attention fusion structure is the DVS fusion feature, and the first output data and the second output data of each of the attention fusion structures are input into a corresponding feature memory structure for data processing. In the feature memory structure, an interaction key vector and an interaction value vector are obtained according to an accumulation result of the first output data and the second output data, an initial interaction vector corresponding to the current attention fusion structure is obtained as an interaction query vector, and an initial value of the initial interaction vector is a preset vector; an interaction feature is obtained according to attention calculation of the interaction query vector, the interaction key vector, and the interaction value vector. The decoding module is configured to input the RGB fusion feature, the DVS fusion feature, and the interaction feature into a decoder to perform decoding and obtain a prediction result corresponding to the current frame.

7. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the multi-modal image fusion decision method of any one of claims 1 to 5 when executing the computer program.

8. A storage medium storing a computer program, characterized by The computer program is executed by the processor to implement the multi-modal image fusion decision method of any one of claims 1 to 5.

Images