Multi-modal image fusion method and system based on traffic scene

Abstract

The invention discloses a multi-modal image fusion method and system based on a traffic scene, and relates to the field of intelligent traffic management, and the method comprises the steps: carrying out the model building based on a multi-scale convolutional neural network and an attention Transformer, extracting the local features of different scales through a multi-scale feature fusion module, and carrying out the fusion of the local features of different scales; and the global feature extraction module is adopted to capture the global features, so that the limitation of multi-scale local feature extraction is solved, the integrity of the global features is reserved, a better effect is achieved in cross-modal image fusion, and the accuracy of multi-modal image fusion is improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent traffic management, and in particular to a multimodal image fusion method and system based on traffic scenarios. Background Technology

[0002] In recent years, high-position video road monitoring technology has developed rapidly. By installing high-position video cameras in traffic scenarios such as roadsides and intersections to collect data, and using visual algorithms to analyze the collected data in a comprehensive manner, it is beneficial to achieve more refined and accurate management of different traffic scenarios. In addition, it can provide support for intelligent transportation systems, traffic safety, data analysis, emergency response, smart city construction, and public transportation optimization, thus playing a positive role in promoting urban traffic management and driving safety.

[0003] Currently, visible light cameras are mainly used in major traffic scenarios to achieve the above functions. However, compared with other spectra, visible light has a significantly narrower imaging range and is only effective in well-lit and high-visibility environments. It may fail in special weather conditions such as nighttime, rain, fog, or obstructions, thus hindering the understanding and analysis of traffic scenarios in special environments. Therefore, various solutions propose using multimodal data from different sensor devices, including visible light cameras, infrared cameras, lidar, and millimeter-wave radar, to analyze roadside parking scenarios. Different modal data can play different advantages under different environmental and weather conditions. The fusion of multiple modal data can provide more data support for the analysis of complex scenarios, which plays an important role in improving the accuracy of traffic scenario analysis.

[0004] Among them, the fusion of visible light cameras and infrared cameras is the most common and relatively lowest cost multi-sensor combination method. Image fusion of the two modalities involves combining the original images captured by visible light and infrared sensors in the same scene into a fused image through comprehensive analysis, selection or enhancement. The fused image can more comprehensively and reliably describe the traffic scene and make full use of the complementarity between the two sensor imaging systems, which is helpful for image processing and visual perception in traffic scenes. Infrared cameras can detect the temperature difference between significant objects and the background, and can detect objects even in poor lighting conditions or even in complete darkness, making it a useful method for traffic scene analysis at night.

[0005] Currently, the main methods for fusing these two modalities of data include traditional methods, CNN-based methods, and attention-based Transformer methods. Traditional methods have limited ability to extract significant features when there are large differences in features between regions, resulting in blurred edges and lack of texture details in the fused image, making it unsuitable for complex fusion scenarios. CNN-based methods perform poorly in extracting global features. Due to the lack of feature extraction methods for different receptive fields, the ability to extract local features is weakened, which may lead to blurring and distortion in the details of the fusion result. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a multimodal image fusion method and system based on traffic scenarios, which can solve the problem of low accuracy in multimodal image fusion based on traffic scenarios.

[0007] To achieve the above objectives, the present invention provides a multimodal image fusion method based on traffic scenarios, the method comprising:

[0008] Local feature information of different modal images is extracted by a pre-set backbone feature extraction network, and branch local feature information is extracted from the feature information of the different modal images by a pre-set branch feature extraction network.

[0009] The local feature information of the different modal images and the local feature information of the branches are fused to obtain the local features of the fused image;

[0010] A pre-set global feature extraction network extracts global feature information from the local features of the fused image and different modal images.

[0011] Furthermore, the step of extracting local feature information of different modal images through a pre-set backbone feature extraction network includes:

[0012] Local feature information of four different modalities at four different scales is extracted by using four convolutional blocks in a pre-set backbone feature extraction network.

[0013] Furthermore, the step of extracting branch local feature information from the feature information of the different modal images through a pre-set branch feature extraction network includes:

[0014] The local feature information of four different modal images at different scales output by the four convolutional blocks in the preset backbone feature extraction network is used as the input of the preset branch feature extraction network to obtain the branch local feature information.

[0015] Further, the step of fusing the local feature information of the different modal images and the branch local feature information to obtain the local features of the fused image includes:

[0016] Alignment of the channel dimension is achieved through convolutional layers;

[0017] At the channel splicing point, local feature information of different modal images and local feature information of branches are spliced ​​and fused in the channel dimension.

[0018] Furthermore, the step of extracting global feature information from the local features of the fused image and different modal images using a pre-set global feature extraction network includes:

[0019] By using pixel-by-pixel addition, local features of images from different modalities and the fused image are fused.

[0020] The fused features are input into the SenseTransformer module of the pre-set global feature extraction network to obtain global features at the first scale;

[0021] The global features at the first scale are input into the block merging layer of the pre-set global feature extraction network, and after feature dimension transformation by the convolutional layer, they are input into the attention Transformer block at the second scale to obtain the global features at the second scale.

[0022] The global features at the second scale are input into the block merging layer of the pre-built global feature extraction network, and after feature dimension transformation by the convolutional layer, they are input into the attention Transformer block at the third scale to obtain the global features at the third scale.

[0023] The global features at the third scale are input into the block merging layer of the pre-built global feature extraction network, and after feature dimension transformation by the convolutional layer, they are input into the attention Transformer block at the fourth scale to obtain the global features at the fourth scale.

[0024] Furthermore, the present invention provides a multimodal image fusion system based on traffic scenarios, the system comprising:

[0025] The extraction module is used to extract local feature information of different modal images through a preset backbone feature extraction network, and to extract branch local feature information from the feature information of the different modal images through a preset branch feature extraction network;

[0026] The fusion module is used to fuse the local feature information of the different modal images and the local feature information of the branches to obtain the local features of the fused image;

[0027] The extraction module is also used to extract global feature information from the local features of the fused image and different modal images through a preset global feature extraction network.

[0028] Furthermore, the extraction module is specifically used to extract local feature information of four different modalities at four different scales through four convolutional blocks in a preset backbone feature extraction network.

[0029] Furthermore, the extraction module is specifically used to take the local feature information of the four different modal images of different scales output by the four convolutional blocks in the preset backbone feature extraction network as the input of the preset branch feature extraction network to obtain branch local feature information.

[0030] Furthermore, the fusion module is specifically used to align the channel dimensions through convolutional layers; and to fuse the local feature information of different modal images and the local feature information of the branches at the channel splicing point.

[0031] Furthermore, the extraction module is specifically used to fuse local features of images from different modalities and the fused image by using a pixel-by-pixel addition operation; input the fused features into the attention Transformer module of a preset global feature extraction network to obtain global features at a first scale; input the global features at the first scale into the block merging layer of the preset global feature extraction network, and after feature dimension transformation by a convolutional layer, input them into the attention Transformer block at a second scale to obtain global features at a second scale; input the global features at the second scale into the block merging layer of the preset global feature extraction network, and after feature dimension transformation by a convolutional layer, input them into the attention Transformer block at a third scale to obtain global features at a third scale; input the global features at the third scale into the block merging layer of the preset global feature extraction network, and after feature dimension transformation by a convolutional layer, input them into the attention Transformer block at a fourth scale to obtain global features at a fourth scale.

[0032] This invention provides a multimodal image fusion method and system based on traffic scenarios. The model is built based on a multi-scale convolutional neural network and attention Transformer. It uses a multi-scale feature fusion module to extract local features at different scales and a global feature extraction module to capture global features. This solves the limitations of multi-scale local feature extraction while preserving the integrity of global features. It achieves better results in cross-modal image fusion, thereby improving the accuracy of multimodal image fusion. Attached Figure Description

[0033] Figure 1 This is a flowchart of a multimodal image fusion method based on traffic scenarios provided by the present invention;

[0034] Figure 2 This is a schematic diagram of a multimodal image fusion system based on traffic scenarios provided by the present invention. Detailed Implementation

[0035] The device structure and implementation of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0036] This invention provides a multimodal image fusion method based on traffic scenarios, such as... Figure 1 As shown, the specific steps include the following:

[0037] 101. Local feature information of different modal images is extracted by a preset backbone feature extraction network, and branch local feature information is extracted from the feature information of the different modal images by a preset branch feature extraction network.

[0038] Specifically, local feature information of four different modalities at four different scales is extracted by four convolutional blocks in the preset backbone feature extraction network. The local feature information of the four different modalities at four different scales output by the four convolutional blocks in the preset backbone feature extraction network is used as the input of the preset branch feature extraction network to obtain branch local feature information.

[0039] For example, specifically, the multi-scale feature extraction and fusion module includes three parts: a backbone feature extraction part, a multi-branch multi-scale feature extraction part, and a feature fusion part. The backbone feature extraction part consists of two identical convolutional neural networks, used to extract features from two different modalities of the image, preventing confusion between the two features. Each convolutional neural network consists of four convolutional blocks, used to extract features at four different scales. Each convolutional block consists of a convolutional layer, a ReLU activation function layer, and a normalization layer. The kernel size of each convolutional layer is 3*3. The multi-branch, multi-scale feature extraction section consists of three branches. The inputs to the first, second, and third branches are the outputs of the first, second, and third convolutional blocks in the main feature extraction section, respectively. Specifically, taking the input of the first branch as an example, the input is the output of the first convolutional block in the main feature extraction section, which consists of features of two modalities. First, the features of the two modalities are concatenated along the channel dimension using a channel concatenation layer, and then fed into a convolutional layer with a 1*1 kernel for further feature extraction and feature dimension transformation, and then fed into the first branch as input. The inputs of the other branches are similar.

[0040] The network structure for each branch is as follows: the first branch consists of three convolutional blocks, the second branch consists of two convolutional blocks, and the third branch consists of one convolutional block. Each convolutional block consists of a convolutional layer, an activation function layer, and a normalization layer. The kernel size of each convolutional layer is 3*3.

[0041] For embodiments of the present invention, before step 101, the following may be included: installation of multimodal sensing devices and collection and data alignment of multimodal data in traffic scenarios to construct a dataset; specifically, multimodal sensing data collection devices, including visible light cameras and infrared cameras, are installed at different traffic locations to acquire multimodal sensing data; the different traffic locations include major urban roads, intersections, roadside parking areas, and parking areas at the entrances of key areas such as schools and hospitals, covering dynamic change data of vehicles, pedestrians, and other targets in the above-mentioned different scenarios at different time periods; because different sensors may have different sensing ranges, resolutions, and sensing capabilities, in order to subsequently fuse the multimodal data, it is necessary to first process the acquired multimodal sensing data packets. Data alignment is performed on RGB and infrared images. This data alignment primarily includes two aspects: temporal and spatial dimensions. Spatial alignment unifies the images acquired by the two modal sensors into the same coordinate system. Temporal alignment addresses the fact that the data acquisition frequencies of the two modal sensors may differ, thus requiring temporal alignment to ensure that the acquired data from both devices represents the same scene and the same moment in perception. Through this data acquisition and alignment, paired visible light and infrared image data are obtained. In addition to self-acquired data, paired multimodal image data from open-source datasets, including but not limited to TNO and RoadScene datasets, can also be used to form the training dataset together with the self-acquired data.

[0042] 102. The local feature information of the different modal images and the local feature information of the branches are fused to obtain the local features of the fused image.

[0043] Specifically, the local feature information of four different modal images at different scales output by the four convolutional blocks in the preset backbone feature extraction network is used as the input of the preset branch feature extraction network to obtain branch local feature information.

[0044] For example, the feature fusion part has five inputs: features from two modal images in the main feature extraction part, and feature outputs from three branches in the multi-branch multi-scale feature extraction part. For the fusion of these five features, a 1*1 convolutional layer is first used to align the channel dimensions, then the five features are spliced ​​and fused in the channel dimensions at the channel splicing point, and finally a 1*1 convolution is used for further feature extraction and feature dimension transformation.

[0045] 103. Extract global feature information from the local features of the fused image and different modal images using a pre-set global feature extraction network.

[0046] Specifically, local features of images from different modalities and the fused image are fused using a pixel-by-pixel addition operation; the fused features are input into the attention Transformer module of a pre-set global feature extraction network to obtain global features at a first scale; the global features at the first scale are input into the block merging layer of the pre-set global feature extraction network, and after feature dimension transformation by a convolutional layer, they are input into the attention Transformer block at a second scale to obtain global features at a second scale; the global features at the second scale are input into the block merging layer of the pre-set global feature extraction network, and after feature dimension transformation by a convolutional layer, they are input into the attention Transformer block at a third scale to obtain global features at a third scale; the global features at the third scale are input into the block merging layer of the pre-set global feature extraction network, and after feature dimension transformation by a convolutional layer, they are input into the attention Transformer block at a fourth scale to obtain global features at a fourth scale.

[0047] For example, this module mainly uses a swin transformer as its core component for global feature extraction. First, it fuses the input images of the two modalities and the fused image from step 2 using a pixel-wise addition operation. Then, it uses a 1*1 convolutional layer for dimensionality transformation, outputting H*W*3, where H and W represent the height and width of the image, respectively, and 3 indicates three channels. Next, it uses a block segmentation layer to divide the H*W*3 image into M*M*C local windows, where M represents the width and height of each block, and C is the number of blocks. Finally, it uses a linear embedding layer to represent HW / M. 2 *M 2 *C feature, where HW / M 2 The total number of windows can be set to H=W=128, C=3, M=4. Then, the features after the linear embedding layer are input into the attention Transformer module to obtain the global features at the first scale. In order to obtain the global features at the second scale, the global features at the first scale are first input into the block merging layer. After the feature dimension is transformed by the convolutional layer, they are input into the attention Transformer block at the second scale to obtain the global features at the second scale. By performing the same operation on the third and fourth scales, global features at four different scales can be obtained in the end.

[0048] The attention transformer module adopts a Swing transformer network structure; the block segmentation layer divides a large image into multiple smaller, fixed-size blocks, which makes subsequent processing and calculation steps more efficient and helps to capture local features in the image; the block merging layer merges adjacent blocks into larger blocks, effectively reducing the number of feature maps, reducing the complexity of subsequent calculations, and promoting information fusion.

[0049] Further, a loss function is constructed, and the model is trained.

[0050] Specifically, the model's loss function consists of two parts: a global feature loss function and a structural similarity loss function, expressed as follows:

[0051] L=αL global +βL ssim

[0052] Among them, L global L represents the global loss function. ssim Let represent the structural similarity loss function; α and β are the weight hyperparameters of the two loss functions, respectively.

[0053] Specifically, the global loss function is used to evaluate the difference between the fused image and the original image at multiple scales, and is expressed as follows:

[0054] Where GF, GA, and GB represent global features at different scales of the fused image, infrared image, and visible light image, respectively, and i represents different scales, i∈[1,..,4], representing the four scales mentioned in step 3; max() represents the maximum operation, in the global loss, the maximum value of the global features of the infrared image and the visible light image is calculated at each scale, and then the difference is calculated with the global features of the fused image;

[0055] The structural similarity loss function aims to ensure that the fused image maintains the structural characteristics of the cross-modal image while enhancing overall quality and fidelity. Specifically, it is expressed as follows:

[0056] Specifically, and Let represent the structural similarity loss between the fused image and the infrared image and the visible light image, respectively, as follows:

[0057]

[0058] Where SSIM() represents the structural similarity calculation operation, I f I represents the fused image. iIndicating an infrared image, I v Represents a visible light image.

[0059] This invention provides a multimodal image fusion method based on traffic scenarios. The method is based on a multi-scale convolutional neural network and attention Transformer to build the model. It uses a multi-scale feature fusion module to extract local features at different scales and a global feature extraction module to capture global features. This solves the limitations of multi-scale local feature extraction while preserving the integrity of global features. It achieves better results in cross-modal image fusion, thereby improving the accuracy of multimodal image fusion.

[0060] As Figure 1 The specific implementation of the method shown in this invention provides a multimodal image fusion system based on traffic scenarios, such as... Figure 2 As shown, the system includes: an extraction module 21, used to extract local feature information of different modal images through a preset backbone feature extraction network, and to extract branch local feature information from the feature information of the different modal images through a preset branch feature extraction network;

[0061] The fusion module 22 is used to fuse the local feature information of the different modal images and the branch local feature information to obtain the local features of the fused image;

[0062] The extraction module 21 is further configured to extract global feature information from the local features of the fused image and different modal images through a preset global feature extraction network.

[0063] Furthermore, the extraction module 21 is specifically used to extract local feature information of four different modalities of four different scales through four convolutional blocks in a preset backbone feature extraction network.

[0064] Furthermore, the extraction module 21 is specifically used to take the local feature information of the four different modal images of different scales output by the four convolutional blocks in the preset backbone feature extraction network as the input of the preset branch feature extraction network to obtain branch local feature information.

[0065] Furthermore, the fusion module 22 is specifically used to align the channel dimensions through convolutional layers; and to fuse the local feature information of different modal images and the local feature information of the branches at the channel splicing point.

[0066] Furthermore, the extraction module 21 is specifically used to fuse local features of images of different modalities and fused images by using pixel-by-pixel addition operations; input the fused features into the attention Transformer module of a preset global feature extraction network to obtain global features at a first scale; input the global features at the first scale into the block merging layer of the preset global feature extraction network, and after feature dimension transformation by a convolutional layer, input them into the attention Transformer block at a second scale to obtain global features at a second scale; input the global features at the second scale into the block merging layer of the preset global feature extraction network, and after feature dimension transformation by a convolutional layer, input them into the attention Transformer block at a third scale to obtain global features at a third scale; input the global features at the third scale into the block merging layer of the preset global feature extraction network, and after feature dimension transformation by a convolutional layer, input them into the attention Transformer block at a fourth scale to obtain global features at a fourth scale.

[0067] This invention provides a multimodal image fusion system based on traffic scenarios. The system is built on a multi-scale convolutional neural network and attention Transformer. It uses a multi-scale feature fusion module to extract local features at different scales and a global feature extraction module to capture global features. This solves the limitations of multi-scale local feature extraction while preserving the integrity of global features. It achieves better results in cross-modal image fusion, thereby improving the accuracy of multimodal image fusion.

[0068] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0069] In the above detailed description, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features of the single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, wherein each claim stands alone as a preferred embodiment of the invention.

[0070] The disclosed embodiments have been described above to enable any person skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit and scope of this disclosure. Therefore, this disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0071] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," as interpreted when used as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

[0072] Those skilled in the art will also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the functions of the various illustrative components, units, and steps described above have been generally described. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functions using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention.

[0073] The various illustrative logic blocks or units described in the embodiments of this invention can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0074] The steps of the methods or algorithms described in the embodiments of this invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0075] In one or more exemplary designs, the functions described in the embodiments of the present invention can be implemented in hardware, software, firmware, or any combination of these three. If implemented in software, these functions can be stored on a computer-readable medium or transmitted on a computer-readable medium in the form of one or more instructions or code. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. Storage media can be any available media that can be accessed by a general-purpose or special-purpose computer. For example, such computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program code in the form of instructions or data structures and other forms that can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Furthermore, any connection can be suitably defined as a computer-readable medium, for example, if the software is transmitted from a website, server or other remote resource via a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disks and discs mentioned include compressed disks, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs. Disks typically copy data magnetically, while disks typically copy data optically using lasers. Combinations of the above can also be contained in computer-readable media.

[0076] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal image fusion method based on traffic scenes, characterized in that, The method includes: Local feature information of different modal images is extracted by a pre-set backbone feature extraction network, and branch local feature information is extracted from the feature information of the different modal images by a pre-set branch feature extraction network. The local feature information of the different modal images and the local feature information of the branches are fused to obtain the local features of the fused image; A pre-set global feature extraction network extracts global feature information from the local features of the fused image and different modal images.

2. The multimodal image fusion method based on traffic scenes according to claim 1, characterized in that, The step of extracting local feature information of different modal images through a pre-set backbone feature extraction network includes: Local feature information of four different modalities at four different scales is extracted by using four convolutional blocks in a pre-set backbone feature extraction network.

3. The multimodal image fusion method based on traffic scenes according to claim 2, characterized in that, The step of extracting branch local feature information from the feature information of the different modal images through a pre-set branch feature extraction network includes: The local feature information of four different modal images at different scales output by the four convolutional blocks in the preset backbone feature extraction network is used as the input of the preset branch feature extraction network to obtain the branch local feature information.

4. The multimodal image fusion method based on traffic scenes according to claim 1, characterized in that, The step of fusing the local feature information of the different modal images and the branch local feature information to obtain the local features of the fused image includes: Alignment of the channel dimension is achieved through convolutional layers; At the channel splicing point, local feature information of different modal images and local feature information of branches are spliced ​​and fused in the channel dimension.

5. The multimodal image fusion method based on traffic scenes according to claim 1, characterized in that, The step of extracting global feature information from the local features of the fused image and different modal images through a pre-set global feature extraction network includes: By using pixel-by-pixel addition, local features of images from different modalities and the fused image are fused. The fused features are input into the SenseTransformer module of the pre-set global feature extraction network to obtain global features at the first scale; The global features at the first scale are input into the block merging layer of the pre-set global feature extraction network, and after feature dimension transformation by the convolutional layer, they are input into the attention Transformer block at the second scale to obtain the global features at the second scale. The global features at the second scale are input into the block merging layer of the pre-built global feature extraction network, and after feature dimension transformation by the convolutional layer, they are input into the attention Transformer block at the third scale to obtain the global features at the third scale. The global features at the third scale are input into the block merging layer of the pre-built global feature extraction network, and after feature dimension transformation by the convolutional layer, they are input into the attention Transformer block at the fourth scale to obtain the global features at the fourth scale.

6. A multimodal image fusion system based on traffic scenes, characterized in that, The system includes: The extraction module is used to extract local feature information of different modal images through a preset backbone feature extraction network, and to extract branch local feature information from the feature information of the different modal images through a preset branch feature extraction network; The fusion module is used to fuse the local feature information of the different modal images and the local feature information of the branches to obtain the local features of the fused image; The extraction module is also used to extract global feature information from the local features of the fused image and different modal images through a preset global feature extraction network.

7. A multimodal image fusion system based on traffic scenes according to claim 6, characterized in that, The extraction module is specifically used to extract local feature information of four different modalities at four different scales through four convolutional blocks in a pre-set backbone feature extraction network.

8. A multimodal image fusion system based on traffic scenes according to claim 7, characterized in that, The extraction module is further configured to use the local feature information of four different modal images of different scales output by the four convolutional blocks in the preset backbone feature extraction network as the input of the preset branch feature extraction network to obtain branch local feature information.

9. A multimodal image fusion system based on traffic scenes according to claim 6, characterized in that, The fusion module is specifically used to align the channel dimensions through convolutional layers; and to fuse the local feature information of different modal images and the local feature information of the branches at the channel splicing point.

10. A multimodal image fusion system based on traffic scenes according to claim 6, characterized in that, The extraction module is further configured to fuse local features of images from different modalities and the fused image by using a pixel-by-pixel addition operation; input the fused features into the attention Transformer module of a preset global feature extraction network to obtain global features at the first scale; input the global features at the first scale into the block merging layer of the preset global feature extraction network, and after feature dimension transformation by a convolutional layer, input them into the attention Transformer block at the second scale to obtain global features at the second scale; The global features at the second scale are input into the block merging layer of the pre-built global feature extraction network, and after feature dimension transformation by the convolutional layer, they are input into the attention Transformer block at the third scale to obtain the global features at the third scale. The global features at the third scale are input into the block merging layer of the pre-built global feature extraction network, and after feature dimension transformation by the convolutional layer, they are input into the attention Transformer block at the fourth scale to obtain the global features at the fourth scale.

Images