Multimodal image fusion detection method and device for train operating environment perception

Abstract

The present invention discloses a multimodal image fusion detection method and device for train operation environment perception, which relates to the field of image processing technology. The method comprises: acquiring image data of different modalities of railway scenes under various environmental conditions, wherein the image data of different modalities include visible light image data and infrared image data, and the image data of different modalities correspond to the same target and are collected at the same time; performing feature extraction on the image data of different modalities to obtain corresponding modal feature information, and obtaining feature information after interactive fusion of features of the image data of different modalities through cross-fusion attention; performing image fusion and target detection on the feature information after interactive fusion of features of the image data of different modalities to obtain a fused image and detection results of the target on the fused image. The present invention solves the problems of the prior art in processing multimodal images, such as the separation of fusion and detection, and insufficient feature interaction.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multimodal image fusion detection method and device for train operating environment perception. Background Art

[0002] With the rapid development of computer vision and artificial intelligence technologies, multimodal image fusion detection technology has found extensive application in railway traffic safety monitoring, intelligent perimeter intrusion protection, and equipment status inspection along the railway. Infrared and visible light images, as two complementary modalities, each offer distinct imaging advantages. Infrared images can capture thermal radiation information in complex lighting or nighttime environments, while visible light images provide rich texture and color details. By fusing image information from these two modalities, the robustness and accuracy of tasks such as target detection and recognition can be significantly improved.

[0003] In the railway sector, existing multimodal image processing methods generally optimize for only a single task. Some technologies focus solely on the fusion quality of multimodal images, lacking specialized design for detection tasks; others construct network models solely for multimodal target detection, failing to fully utilize the fusion information of multimodal features. Such methods often fail to establish an effective connection between fusion and detection tasks, making it difficult to fully leverage the complementary advantages of multimodal information. Furthermore, existing multi-task joint optimization methods, when processing multimodal images, generally suffer from insufficient feature interaction, significant interference between tasks, and unstable training, impacting the overall performance of railway driving environment fusion detection systems. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a multimodal image fusion detection method and device for train operation environment perception, aiming to solve the problems commonly existing in the prior art when processing multimodal images, such as insufficient feature interaction and large interference between tasks.

[0005] In one aspect, the present invention proposes a multimodal image fusion detection method for train operating environment perception, the method comprising:

[0006] Acquire image data of different modalities of railway scenes under various environmental conditions, wherein the image data of different modalities include visible light image data and infrared image data, and the image data of different modalities correspond to the same target and are collected at the same time;

[0007] Feature extraction is performed on image data of different modalities to obtain corresponding modality feature information, and feature information obtained by interactive fusion of features of image data of different modalities is obtained through cross-fusion attention;

[0008] The feature information after the feature interaction and fusion of image data of different modalities is respectively subjected to image fusion and object detection to obtain a fused image and the detection result of the object on the fused image.

[0009] Further, in the above multi-modal image fusion detection method for train operation environment perception, before the step of respectively extracting features from image data of different modalities to obtain corresponding modal feature information and obtaining the feature information after the feature interaction and fusion of image data of different modalities through cross-fusion attention, it further includes:

[0010] Judging whether the visible light image data and the infrared image data are aligned;

[0011] If so, execute the step of respectively extracting features from image data of different modalities to obtain corresponding modal feature information;

[0012] If not, perform a registration operation on the visible light image data and the infrared image data to complete spatial alignment and then execute the step of respectively extracting features from image data of different modalities to obtain corresponding modal feature information;

[0013] Among them, when S≥r, the visible light image data and the infrared image data are aligned, and when S<r, the visible light image data and the infrared image data are not aligned, where r represents a preset threshold for modal alignment;

[0014]

[0015]

[0016]

[0017]

[0018] The expression for performing a registration operation on the visible light image data and the infrared image data to complete spatial alignment is:

[0019] [[ID=3 forty]]

[0020]

[0021]

[0022] Among them, represents image data of different modalities, represents infrared image data, represents visible light image data, represents a shared feature extraction network, represents infrared image features, represents visible light image features, and S represents and The similarity of Sim Represents calculation and Similarity operation, represents the transpose of infrared image features, represents the norm, Represents a spatial correction alignment operation, represents the corrected visible light image, represents matrix transformation, represents the horizontal coordinate of the image pixel, represents the vertical coordinate of the image pixel, Represents the image pair after correction alignment.

[0023] Furthermore, in the above-mentioned multimodal image fusion detection method for train operation environment perception, the expression of the feature information obtained by interactive fusion of features of image data of different modalities through cross-fusion attention is:

[0024]

[0025]

[0026]

[0027]

[0028]

[0029] in, represents the visible light modal feature map, represents the infrared modal feature map, represents the learnable weight matrix used to generate the query vector, represents the learnable weight matrix used to generate the key vector, represents the visible light modality query vector, represents the infrared modality query vector, represents matrix multiplication, represents the transpose of the infrared modal bond vector, represents the transpose of the visible light modal bond vector, represents the dimension of the query or key vector, Indicates the normalization of attention weights Softmax function, represents the cross attention weight of infrared to visible light, represents the cross attention weight of visible light to infrared, represents a 1×1 convolution operation, represents the batch normalization operation, represents the activation function, represents the visible light characteristics after interactive enhancement, represents the infrared features after interactive enhancement, Represents the feature information after the final interactive fusion.

[0030] Furthermore, in the multimodal image fusion detection method for train operating environment perception, the step of obtaining feature information obtained by interactive fusion of features of image data of different modalities through cross-fusion attention further includes:

[0031] The features of image data of different modalities are extracted and reconstructed at different granularity levels: object level, region level and pixel level.

[0032] Among them, the target level is used to enhance high-level semantic features and capture the overall spatial distribution characteristics of the target area, the region level is used to focus on local structure and edge information, and the pixel level is used to retain the texture and details in the original image.

[0033] Furthermore, in the multimodal image fusion detection method for train operating environment perception, the steps of performing image fusion and target detection on the feature information obtained by interactively fusing features of image data of different modalities to obtain a fused image and a detection result of a target on the fused image include:

[0034] A gradient alignment mechanism is introduced during the training of image fusion and object detection tasks. This mechanism orthogonalizes the gradients of the fusion and object detection tasks to ensure that the two tasks have the same direction during the shared parameter update process.

[0035] Among them, the expression of the gradient alignment loss function is:

[0036]

[0037]

[0038]

[0039] in, represents the shared parameter set in the fusion task and the target detection task, Indicates the parameters The gradient operator, Represents the fusion task loss function parameter The gradient, Represents the target detection task loss function parameter The gradient, Represents the fusion task loss function parameter The transpose of the gradient of represents the gradient alignment loss function.

[0040] Furthermore, in the above-mentioned multimodal image fusion detection method for train operating environment perception, the loss function expression of the image fusion task is:

[0041]

[0042]

[0043]

[0044]

[0045] in, represents a visible light image, represents an infrared image, represents the generated fused image, represents the image structure similarity index, represents the structural similarity loss, represents the pixel loss in the target area, Represents the pixel loss in the background area, The mask representing the target region, Represents the maximum value at the corresponding pixel position, represents the L1 norm, represents the mean value at the corresponding pixel position, represents the weighted coefficient of structural similarity loss, Represents the weighted coefficient of pixel loss in the target area, represents the fusion loss.

[0046] Furthermore, in the above-mentioned multimodal image fusion detection method for train operating environment perception, the expression of the target detection task loss function is:

[0047]

[0048] in, Indicates the position of the detection box after prediction and denoising in the detection subnetwork, represents the noisy detection box coordinate set, represents the diffusion time step, Represents the true annotation coordinates of the detection box, represents the L2 norm squared.

[0049] Another object of the present invention is to provide a multimodal image fusion detection device for train operating environment perception, the device comprising:

[0050] An acquisition module is used to acquire image data of different modes of railway scenes under various environmental conditions, wherein the image data of different modes include visible light image data and infrared image data, and the image data of different modes correspond to the same target and are acquired at the same time;

[0051] The fusion module is used to extract features from image data of different modalities respectively to obtain corresponding modal feature information, and obtain feature information after interactive fusion of features of image data of different modalities through cross-fusion attention;

[0052] The detection module is used to perform image fusion and target detection on the feature information after interactive fusion of features of image data of different modalities, so as to obtain the detection results of the fused image and the target on the fused image.

[0053] Another object of the present invention is to provide a readable storage medium having a computer program stored thereon, wherein the program implements the steps of the above method when executed by a processor.

[0054] Another object of the present invention is to provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0055] The present invention obtains image data of different modalities of railway scenes under various environmental conditions, wherein the image data of different modalities include visible light image data and infrared image data, and the image data of different modalities correspond to the same target and are collected at the same time; performs feature extraction on the image data of different modalities respectively to obtain corresponding modal feature information, and obtains feature information after interactive fusion of features of image data of different modalities through cross-fusion attention; performs image fusion and target detection on the feature information after interactive fusion of features of image data of different modalities respectively to obtain fused images and detection results of targets on the fused images. The image fusion and target detection tasks are performed synchronously through the end-to-end structure, effectively improving the utilization efficiency of multimodal images and the target recognition accuracy. It solves the problems of insufficient feature interaction and large interference between tasks in the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Schematic diagram of the overall network structure of a multimodal image fusion detection method for train operating environment perception in one embodiment of the present invention;

[0057] Figure 2 This is a flow chart of a multimodal image fusion detection method for train operating environment perception in the first embodiment of the present invention;

[0058] Figure 3 This is a structural block diagram of a multimodal image fusion detection device for train operating environment perception in the third embodiment of the present invention.

[0059] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0060] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The drawings illustrate several embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present invention.

[0061] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be an intermediate element. When an element is referred to as being "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0063] In related technologies, multi-task joint modeling methods are usually used to achieve collaborative optimization of image fusion and target detection tasks at the same time. By constructing a joint framework for fusion images and detection networks, the ability to express features can be enhanced, and the detection accuracy and fusion image quality can be improved. However, to achieve effective multi-task collaboration, it is necessary to design a network structure with fine feature expression and task decoupling capabilities, so that the two tasks of fusion and detection do not interfere with each other and can be collaboratively optimized based on shared features. This requires the algorithm to not only accurately capture the significant information of images of different modalities, but also to reasonably regulate the gradient influence of each task to avoid performance degradation due to gradient conflicts or task-dominant effects.

[0064] See also Figure 1 , shown is a schematic diagram of the overall network structure of the multimodal image fusion detection method for train operating environment perception proposed in one embodiment of the present invention, which mainly includes a backbone network, an image fusion subnetwork and a target detection subnetwork, and the overall network can be trained accordingly to obtain the final multimodal image fusion detection model for multimodal image fusion and detection.

[0065] Before the visible light modal image and the infrared image are input into the backbone network, they will first pass through the modality alignment module to align the visible light modal image and the infrared image; further, the aligned image will pass through the feature extraction module in the backbone network to obtain the feature information of different modalities; the extracted multimodal features will be input into the cross-fusion attention to perform semantic interaction between modalities and enhance salient areas, and will be fused into a unified feature representation through the feature fusion module; the fused features will be input into the image fusion subnetwork and the target detection subnetwork respectively, and high-quality fused images and accurate target detection results on the fused images will be output simultaneously.

[0066] Specifically, the modality alignment module is used to address the spatial displacement or geometric mismatch between infrared and visible light images. It determines whether the input images are aligned through feature matching and transformation estimation. If not, it uses image registration technology to align the two modal images so that they remain consistent in spatial position, laying the foundation for subsequent feature extraction and fusion.

[0067] The feature extraction module is used to extract multi-scale and multi-semantic features from infrared and visible light images respectively. It consists of two parallel deep neural network branches. Through specific parameter design, it fully models the information of different modalities and retains key information such as edges, textures, heat sources, and colors.

[0068] Cross-fusion attention is used to establish semantic connections between the two modalities and achieve deep information interaction between the modalities. By using the attention mechanism, the features of one modality are used as queries and the features of the other modality are used as keys for correlation calculation, thereby enhancing the response of complementary areas and improving the discriminability and expressiveness of fused features.

[0069] The feature fusion module uniformly encodes and fuses feature maps from infrared and visible light modalities. Fusion methods include weighted summation, splicing, and dynamic channel fusion. It also combines residual structure and scale alignment mechanisms to ensure that different modalities can be collaboratively represented at the spatial distribution and semantic levels, providing a unified and high-quality feature foundation for subsequent tasks.

[0070] The image fusion subnetwork generates a fused image based on the fused features. This image retains both the details of the visible light image and the heat source information of the infrared image. It usually adopts a decoder structure and combines multiple loss functions such as structural similarity loss and texture loss to optimize the reconstruction effect, ensuring that the fused image has good performance in terms of clarity, contrast, and object prominence.

[0071] The target detection subnetwork is used to identify and locate targets on fused images or fused features, output the target category and location information, use a lightweight detection network to improve the detection accuracy of small targets, occluded targets, and rotated targets, and coordinate optimization with the image fusion module to enhance overall performance.

[0072] Example 1

[0073] See also Figure 2 , which shows a multimodal image fusion detection method for train running environment perception in the first embodiment of the present invention, and the method includes steps S10 to S12.

[0074] Step S10 , acquiring image data of different modalities of railway scenes under various environmental conditions, wherein the image data of different modalities include visible light image data and infrared image data, and the image data of different modalities correspond to the same target and are collected at the same time.

[0075] The embodiments of the present invention are primarily used in railway scenarios. Image data of different modalities includes visible light image data and infrared image data collected simultaneously for the same target ahead of a train. By collecting real-time information ahead of the railway during train operation and performing corresponding image analysis and detection, tasks such as railway traffic safety monitoring, intelligent perimeter intrusion protection, and equipment status inspection along the line can be performed based on the results of the analysis and detection. Specifically, the various environmental conditions include sunny, rainy, snowy, foggy, strongly exposed, and nighttime conditions.

[0076] Step S11, feature extraction is performed on image data of different modalities to obtain corresponding modal feature information, and feature information obtained by interactive fusion of features of image data of different modalities is obtained through cross-fusion attention;

[0077] Among them, through the two parallel deep neural network branches in the overall network architecture, namely the feature extraction module, feature extraction is performed on the image data of different modalities respectively to obtain the corresponding modal feature information. The different modal feature information is then input into the feature fusion module, and deep information interaction between modalities is realized based on cross-fusion attention. Among them, the feature fusion module adopts a multi-branch structure, from a coarse to fine perspective, and extracts and reconstructs each modal feature at different granularity levels from the target level, region level and pixel level. Specifically, after the image data of different modalities interact through cross-attention, the target-level branch in the feature fusion module strengthens the high-level semantic features and captures the overall spatial distribution characteristics of the target area. The region-level branch focuses on the local structure and edge information, while the pixel-level branch is used to retain the texture and details in the original image.

[0078] Specifically, the expression of the feature information obtained by interactive fusion of features of image data of different modalities through cross fusion attention is:

[0079]

[0080]

[0081]

[0082]

[0083]

[0084] in, represents the visible light modal feature map, represents the infrared modal feature map, represents the learnable weight matrix used to generate the query vector, represents the learnable weight matrix used to generate the key vector, represents the visible light modality query vector, represents the infrared modality query vector, represents matrix multiplication, represents the transpose of the infrared modal bond vector, represents the transpose of the visible light modal bond vector, represents the dimension of the query or key vector, Indicates the normalization of attention weights Softmax function, represents the cross attention weight of infrared to visible light, represents the cross attention weight of visible light to infrared, represents a 1×1 convolution operation, represents the batch normalization operation, represents the activation function, represents the visible light characteristics after interactive enhancement, represents the infrared features after interactive enhancement, Represents the feature information after the final interactive fusion.

[0085] In addition, in some optional embodiments of the present invention, before the step of extracting features from image data of different modalities to obtain corresponding modal feature information, and obtaining feature information obtained by interactive fusion of features of image data of different modalities through cross-fusion attention, the step further includes:

[0086] Determining whether the visible light image data and the infrared image data are aligned;

[0087] If so, perform the step of separately extracting features from image data of different modalities to obtain corresponding modality feature information;

[0088] If not, perform a registration operation on the visible light image data and the infrared image data to complete spatial alignment, and then perform the step of separately extracting features from image data of different modalities to obtain corresponding modality feature information.

[0089] Among them, the modality alignment module is used to align image data of different modalities to solve the spatial displacement or geometric mismatch problem between infrared images and visible light images. Specifically, when S≥r, the visible light image data and the infrared image data are aligned; when S<r, the visible light image data and the infrared image data are not aligned, where r represents a preset modality alignment threshold;

[0090]

[0091]

[0092]

[0093]

[0094] The expression for performing a registration operation on the visible light image data and the infrared image data to complete spatial alignment is:

[0095]

[0096]

[0097]

[0098] Among them, represents image data of different modalities, represents infrared image data, represents visible light image data, represents a shared feature extraction network, represents infrared image features, represents visible light image features, S represents and the similarity of, Sim represents calculating and the similarity operation of, represents the transpose of infrared image features, represents the norm, <000034​​​​​​represents the horizontal coordinate of the image pixel, represents the vertical coordinate of the image pixel, Represents the image pair after correction alignment.

[0099] In step S12, image fusion and target detection are performed on the feature information obtained by interactively fusing the features of the image data of different modalities, so as to obtain a fused image and a detection result of the target on the fused image.

[0100] Among them, the image fusion sub-network encodes the feature information after interactive fusion of features to generate the final fused image. The obtained fused image has advantages in structural fidelity, target saliency and information integrity, and can take into account the feature expression of infrared and visible light modalities at the same time; the deep semantic feature information of the fused image is input into the target detection sub-network. The target detection sub-network interacts the feature information after interactive fusion of different modal features with the deep semantic feature information of the fused image to obtain accurate target detection results on the fused image, including detection box position and category label.

[0101] Specifically, the object detection subnetwork employs a diffusion-based detection box generation strategy, modeling the object detection task as a regression process with gradual denoising. Initial detection boxes are generated using Gaussian noise, and the network, under guidance, gradually recovers the true object location and category, achieving coarse-to-fine optimization of detection results and improving detection accuracy and robustness.

[0102] In summary, the multimodal image fusion detection method for train operation environment perception in the above embodiment of the present invention obtains image data of different modes of railway scenes under various environmental conditions, wherein the image data of different modes include visible light image data and infrared image data, and the image data of different modes correspond to the same target and are collected at the same time; feature extraction is performed on the image data of different modes respectively to obtain corresponding modal feature information, and feature information obtained by interactive fusion of features of image data of different modes is obtained through cross-fusion attention; image fusion and target detection are performed on the feature information after interactive fusion of features of image data of different modes respectively to obtain the fused image and the detection results of the target on the fused image. The image fusion and target detection tasks are performed synchronously through the end-to-end structure, which effectively improves the utilization efficiency of multimodal images and the target recognition accuracy. It solves the problems of insufficient feature interaction and large interference between tasks in the existing technology.

[0103] Example 2

[0104] This embodiment also proposes a multimodal image fusion detection method for train operating environment perception. The multimodal image fusion detection method for train operating environment perception in this embodiment differs from the multimodal image fusion detection method for train operating environment perception in Example 1 in that:

[0105] During the overall network training process, a gradient alignment mechanism is introduced to ensure synergy between the image fusion task and the object detection task. This mechanism orthogonalizes the gradients of the fusion and detection tasks to resolve gradient conflicts that may arise during multi-task training. This ensures that each task maintains consistent direction during the shared parameter update process, thereby achieving complementary synergy between the tasks and improving overall performance.

[0106] Specifically, the loss function expression of image fusion is:

[0107]

[0108]

[0109]

[0110]

[0111] in, represents a visible light image, represents an infrared image, represents the generated fused image, Represents the image structure similarity index (the closer the value is to 1, the more consistent the structure is), represents the structural similarity loss, represents the pixel loss in the target area, Represents the pixel loss in the background area, The mask representing the target region, Represents the maximum value at the corresponding pixel position, represents the L1 norm (summing the element-wise absolute values), represents the mean value at the corresponding pixel position, represents the weighted coefficient of structural similarity loss, Represents the weighted coefficient of pixel loss in the target area, represents the fusion loss.

[0112] The expression of the detection loss function is:

[0113]

[0114] in, Indicates the position of the detection box after prediction and denoising in the detection subnetwork, represents the noisy detection box coordinate set (the t-th step state in the diffusion process), represents the diffusion time step (used to condition the network prediction), Represents the true annotation coordinates of the detection box, Represents the L2 norm square (used to calculate the Euclidean distance error between the predicted detection box and the true detection box).

[0115] The expression of the gradient alignment loss function is:

[0116]

[0117]

[0118]

[0119] in, represents the set of shared parameters in the multi-task network, Indicates the parameters The gradient operator, Represents the fusion task loss function parameter The gradient, Represents the detection task loss function parameter The gradient, Represents the fusion task loss function parameter The transpose of the gradient of represents the gradient alignment loss function.

[0120] The expression of the total loss function is:

[0121]

[0122] in, represents the weighted coefficient of fusion loss, represents the weighted coefficient of detection loss, represents the weight coefficient of gradient alignment loss, represents the total loss function.

[0123] In summary, the multimodal image fusion detection method for train operation environment perception in the above embodiment of the present invention obtains image data of different modes of railway scenes under various environmental conditions, wherein the image data of different modes include visible light image data and infrared image data, and the image data of different modes correspond to the same target and are collected at the same time; feature extraction is performed on the image data of different modes respectively to obtain corresponding modal feature information, and feature information obtained by interactive fusion of features of image data of different modes is obtained through cross-fusion attention; image fusion and target detection are performed on the feature information after interactive fusion of features of image data of different modes respectively to obtain the fused image and the detection results of the target on the fused image. The image fusion and target detection tasks are performed synchronously through the end-to-end structure, which effectively improves the utilization efficiency of multimodal images and the target recognition accuracy. It solves the problems of insufficient feature interaction and large interference between tasks in the existing technology.

[0124] Example 3

[0125] See also Figure 3 , shown is a multimodal image fusion detection device for train running environment perception proposed in the third embodiment of the present invention, the device comprising:

[0126] An acquisition module 100 is configured to acquire image data of different modalities of a railway scene under various environmental conditions, wherein the image data of different modalities include visible light image data and infrared image data, and the image data of different modalities correspond to the same target and are acquired at the same time;

[0127] The fusion module 200 is used to extract features from image data of different modalities to obtain corresponding modal feature information, and obtain feature information obtained by interactive fusion of features of image data of different modalities through cross-fusion attention;

[0128] The detection module 300 is used to perform image fusion and target detection on the feature information obtained by interactively fusing features of image data of different modalities, so as to obtain a fused image and detection results of targets on the fused image.

[0129] The functions or operation steps implemented when the above modules are executed are substantially the same as those in the above method embodiments and will not be described in detail here.

[0130] Example 4

[0131] Another aspect of the present invention further provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the above-mentioned embodiments 1 to 2.

[0132] Example 5

[0133] On the other hand, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the program, the steps of the method described in any one of the above-mentioned embodiments 1 to 2 are implemented.

[0134] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] Those skilled in the art will appreciate that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device), or for use in conjunction with such instruction execution system, apparatus, or device. For purposes of this specification, a "computer-readable storage medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device.

[0136] More specific examples (a non-exhaustive list) of computer-readable storage media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable storage medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0137] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the aforementioned embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following technologies known in the art may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0138] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0139] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A multimodal image fusion detection method for train operating environment perception, characterized by: The method includes: Obtain image data of different modalities in a railway scene under various environmental conditions. Among them, the image data of different modalities includes visible light image data and infrared image data, and the image data of different modalities corresponds to the same target and is collected at the same time; Extract features from the image data of different modalities respectively to obtain corresponding modality feature information, and obtain the feature information after feature interaction and fusion of the image data of different modalities through cross-fusion attention; The expression for obtaining the feature information after feature interaction and fusion of the image data of different modalities through cross-fusion attention is: in, represents the visible light modal feature map, represents the infrared modal feature map, represents the learnable weight matrix used to generate the query vector, represents the learnable weight matrix used to generate the key vector, represents the visible light modality query vector, represents the infrared modality query vector, represents matrix multiplication, represents the transpose of the infrared modal bond vector, represents the transpose of the visible light modal bond vector, represents the dimension of the query or key vector, Indicates the normalization of attention weights Softmax function, represents the cross attention weight of infrared to visible light, represents the cross attention weight of visible light to infrared, represents a 1×1 convolution operation, represents the batch normalization operation, represents the activation function, represents the visible light characteristics after interactive enhancement, represents the infrared features after interactive enhancement, Represents the feature information after the final interactive fusion; Respectively perform image fusion and target detection on the feature information after feature interaction and fusion of the image data of different modalities to obtain a fused image and the detection result of the target on the fused image. Specifically: [[ID=⑥]]Introduce a gradient alignment mechanism during the training of the image fusion task and the target detection task. The gradient alignment mechanism ensures that the fusion task and the target detection task have the same direction during the shared parameter update process by orthogonalizing the gradients of the fusion task and the target detection task; [[ID=⑦]]Among them, the expression of the gradient alignment loss function is: in, represents the shared parameter set in the fusion task and the target detection task, Indicates the parameters The gradient operator, Represents the fusion task loss function parameter The gradient, Represents the target detection task loss function parameter The gradient, Represents the fusion task loss function parameter The transpose of the gradient of represents the gradient alignment loss function.

2. The multimodal image fusion detection method for train operating environment perception according to claim 1 is characterized in that: [[ID=⑧]]Before the step of extracting features from the image data of different modalities respectively to obtain corresponding modality feature information, and obtaining the feature information after feature interaction and fusion of the image data of different modalities through cross-fusion attention, it further includes: [[ID=⑨]]Judge whether the visible light image data and the infrared image data are aligned; [[ID=⑩]]If so, execute the step of extracting features from the image data of different modalities respectively to obtain corresponding modality feature information; [[ID=⑪]]If not, perform a registration operation on the visible light image data and the infrared image data to complete the spatial alignment, and then execute the step of extracting features from the image data of different modalities respectively to obtain corresponding modality feature information; [[ID=⑫]]Among them, when S≥r, the visible light image data and the infrared image data are aligned, and when S<r, the visible light image data and the infrared image data are not aligned. r represents the preset modality alignment threshold; [[ID=⑬]]The expression for performing a registration operation on the visible light image data and the infrared image data to complete the spatial alignment is: in, Represents image data of different modalities, represents infrared image data, Represents visible light image data, represents the shared feature extraction network, Indicates the infrared image features, Represents the visible light image feature, S represents and The similarity of [[ID=⑭]]Sim Represents calculation and Similarity operation, represents the transpose of infrared image features, represents the norm, Represents a spatial correction alignment operation, represents the corrected visible light image, represents matrix transformation, represents the horizontal coordinate of the image pixel, represents the vertical coordinate of the image pixel, Represents the image pair after correction alignment.

3. The multimodal image fusion detection method for train operating environment perception according to claim 1 is characterized in that: [[ID=⑮]]The step of obtaining the feature information after feature interaction and fusion of the image data of different modalities through cross-fusion attention further includes: [[ID=⑯]]Perform modality feature extraction and reconstruction on the features of the image data of different modalities at different granularity levels from three granularities: target level, region level, and pixel level; [[ID=⑰]]Among them, the target level is used to strengthen high-level semantic features and capture the overall spatial distribution characteristics of the target area, the region level is used to focus on local structures and edge information, and the pixel level is used to retain the texture and details in the original image.

4. The multimodal image fusion detection method for train operating environment perception according to claim 1 is characterized in that: [[ID=⑱]]The expression of the loss function of the image fusion task is: in, represents a visible light image, represents an infrared image, represents the generated fused image, represents the image structure similarity index, represents the structural similarity loss, represents the pixel loss in the target area, Represents the pixel loss in the background area, The mask representing the target region, Represents the maximum value at the corresponding pixel position, represents the L1 norm, represents the mean value at the corresponding pixel position, represents the weighted coefficient of structural similarity loss, Represents the weighted coefficient of pixel loss in the target area, represents the fusion loss.

5. The multimodal image fusion detection method for train operating environment perception according to claim 4 is characterized in that: [[ID=⑲]]The expression of the loss function of the target detection task is: in, Indicates the position of the detection box after prediction and denoising in the detection subnetwork, represents the noisy detection box coordinate set, represents the diffusion time step, Represents the true annotation coordinates of the detection box, represents the L2 norm squared.

6. A multimodal image fusion detection device for train operating environment perception, characterized in that: [[ID=⑳]]A device for implementing the multi-modal image fusion detection method for train operation environment perception according to any one of claims 1 to 5, the device includes: An acquisition module is used to acquire image data of different modalities of railway scenes under various environmental conditions, wherein the image data of different modalities include visible light image data and infrared image data, and the image data of different modalities correspond to the same target and are acquired at the same time; The fusion module is used to extract features from image data of different modalities respectively to obtain corresponding modal feature information, and obtain feature information after interactive fusion of features of image data of different modalities through cross-fusion attention; The detection module is used to perform image fusion and target detection on the feature information after interactive fusion of features of image data of different modalities, so as to obtain the detection results of the fused image and the target on the fused image.

7. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

8. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the steps of the method according to any one of claims 1 to 5 are implemented when the processor executes the program.

Images