A dynamic weight distribution multi-modal fusion method and device based on character guidance

Abstract

The application discloses a kind of dynamic weight distribution multimodal fusion method and device based on character guide, it is related to machine vision and image processing technical field.The method includes: the semantic similarity of character query and multi-source multi-modal input image is calculated, semantic attention map is generated, each modal multi-scale feature is obtained by feature extraction network, feature importance map is obtained by gradient weighted class activation mapping, dynamic weighting map is generated in combination with semantic attention map, modal fusion and scale fusion are sequentially carried out, and the image after multi-scale multi-modal fusion is obtained.The application can realize the accurate binding of character query and multi-source multi-modal image semantics, locate the semantic region related to query in image, dynamically calculate the weight of each source image in different scales and different regions, and realize adaptive image fusion of semantic perception.

Description

Technical Field

[0001] This invention relates to the field of machine vision and image processing technology, and in particular to a text-guided dynamic weight allocation multimodal fusion method and apparatus. Background Technology

[0002] Image fusion technology, as one of the core technologies in machine vision, aims to integrate multi-source image data from different sensors, perspectives, or time periods. It fully leverages the complementary and redundant information of each source image to generate a fused image that surpasses single-source images in terms of information integrity, detail richness, and visual effect. This technology has been widely applied in many key fields such as remote sensing mapping, medical diagnosis, security monitoring, autonomous driving, and military reconnaissance. For example, in remote sensing mapping, fusing multispectral and high-resolution images allows for the simultaneous acquisition of spectral attributes and spatial details of ground features. In medical diagnosis, fusing CT and MRI images provides doctors with more comprehensive lesion information, aiding in accurate diagnosis. With the continuous upgrading of application demands in various fields, the requirements for the intelligence and personalization of image fusion technology are increasing. Traditional image fusion methods have significant drawbacks: firstly, fusion rules rely on manual design, lacking semantic understanding of the fusion target and failing to respond to specific user needs; secondly, fixed fusion weights result in poor adaptability to different scenes and source images, easily leading to detail loss and target blurring in complex scenarios. Meanwhile, existing deep learning-based image fusion technologies still have significant shortcomings: most methods rely solely on pixel-level or feature-level low-level information to drive the fusion process, failing to incorporate high-level semantic information and thus unable to achieve dynamic association between "text query and fusion strategy." When users need to adjust fusion requirements based on specific semantic goals, existing methods require retraining the model for that goal or manually adjusting a large number of parameters, resulting in complex operations, poor flexibility, and limited generalization ability, making it difficult to adapt to diverse fusion needs across multiple domains. In existing technologies, the image fusion process is disconnected from user semantic needs, making it impossible to intuitively control the fusion results through natural language, and thus failing to meet the diverse and personalized fusion needs in practical applications. Therefore, how to construct a text-guided dynamic weight allocation mechanism to achieve intelligent fusion of semantically aware infrared images with other modalities has become a key problem urgently needing to be solved in the current field of image fusion. Summary of the Invention

[0003] To address the technical problems of existing image fusion technologies, such as the disconnect between the image fusion process and user semantic needs, the inability to intuitively control the fusion result through natural language, and the difficulty in meeting the diverse and personalized fusion requirements in practical applications, this invention provides a text-guided dynamic weight allocation multimodal fusion method and apparatus. The technical solution is as follows:

[0004] On the one hand, a text-guided dynamic weight allocation multimodal fusion method is provided, which is implemented by a text-guided dynamic weight allocation multimodal fusion device. The method includes: S1: Acquire multi-source, multi-modal input images, perform preprocessing, and obtain a standardized image set; S2: Receive text queries and perform vectorization processing, extract features and calculate semantic similarity on a standardized image set to obtain a scale-adapted semantic attention map. The vectorization processing includes encoding the text query through a text encoder. S3: Construct a feature extraction network to extract features from each modality image in the standardized image set, and obtain the features of each modality at different scale levels; S4: For the features of each modality at different scale levels, the gradient weighted class activation mapping method is used to calculate the contribution of each channel to the loss function during backpropagation, obtain the weight of each channel, and then weight each channel to obtain the importance map of each modality at each scale. S5: Using the importance maps of each modality and scale features and the scale-adapted semantic attention map, calculate the dynamic weighted map of each modality and scale features, and finally multiply it with the features of multiple modalities at each scale to obtain the multimodal feature fusion map of different scales. S6: Input the multimodal feature fusion map of different scales into the decoding network, and obtain the multimodal fused image through upsampling, normalization activation and post-processing operations.

[0005] Preferably, the multi-source, multi-modal input images acquired in step S1 are preprocessed to obtain a standardized image set, including: S11: Acquire multi-source image data from different sensors to obtain multi-source, multi-modal input images; S12: The multi-source multimodal input image is uniformly scaled to a preset size using a scaling algorithm to obtain a uniform-size image. The scaling algorithm includes bilinear interpolation or nearest neighbor interpolation methods. The uniform-size image includes a single-channel image and / or a multispectral image. S13: Perform channel alignment on the uniform-size image to obtain a channel-aligned image. The channel alignment includes channel duplication of a single-channel image and selection of key channels in a multispectral image. S14: Based on the noise content of the image, the channel-aligned image is denoised to obtain a denoised image. The denoising process includes noise suppression processing through Gaussian filtering, median filtering and / or adaptive Wiener filtering. S15: Perform grayscale normalization on the pixel values ​​of the denoised image and construct a modal index to obtain a standardized image set. The grayscale normalization process includes mapping the pixel values ​​to a preset range using the min-max normalization method.

[0006] Preferably, the received text query in S2 undergoes vectorization processing, involving feature extraction and semantic similarity calculation on a standardized image set to obtain a scale-adapted semantic attention map. The vectorization processing includes encoding the text query using a text encoder, including: S21: Receive a text query, and based on the Transformer architecture, perform vectorization processing on the text query to generate a text query vector. The vectorization processing includes word segmentation, embedding, and multi-layer encoding processing. The dimension of the text query vector is 768. S22: An image encoder is used to extract features from a standardized image set to obtain multi-scale feature maps. The image encoder is based on a ResNet or Swin Transformer architecture. The multi-scale feature maps contain low-level texture features and high-level semantic features. The high-level semantic features include a first-scale feature map, a second-scale feature map, a third-scale feature map, and / or a fourth-scale feature map. The resolution of the first-scale feature map is set to the initial resolution and no downsampling processing is required. The second-scale feature map is obtained by performing a 2×2 max pooling operation based on the first-scale feature map, and the resolution of the second-scale feature map is 320×256. The third-scale feature map is obtained by performing a 4×4 max pooling operation based on the second-scale feature map, and the resolution of the third-scale feature map is 160×128. The fourth-scale feature map is obtained by performing an 8×8 max pooling operation based on the third-scale feature map, and the resolution of the fourth-scale feature map is 80×64. S23: Based on the cross-attention mechanism, the semantic similarity between the text query vector and the high-level semantic features is calculated to locate the semantic region related to the text query, thereby obtaining a scale-adapted semantic attention map. The dimension of the scale-adapted semantic attention map is consistent with the dimension of the multi-scale feature map. The pixel value range in the semantic attention map is a preset range, and the pixel value is used to represent the correlation between the corresponding region and the text query.

[0007] Preferably, the feature extraction network in S3 extracts features from each modality image in the standardized image set to obtain features of each modality at different scale levels, including: S31: Establish a feature extraction network. The feature extraction architecture of the feature extraction network is based on the improved IVMS-UNet 4-level codec symmetric structure, including channel splitting and dual-branch parallel extraction architecture. The dual-branch parallel extraction architecture includes a first feature extraction branch and a second feature extraction branch. The decoder of the improved IVMS-UNet adopts a dual-channel codec symmetric architecture, which includes 4-level downsampling units and 4-level upsampling units. S32: Based on the feature extraction network, the standardized image set is split according to the number of channels to obtain the first channel data and the second channel data. The splitting includes splitting according to the first and second halves of the number of channels. S33: Input the first channel data into the first feature extraction branch, input the second channel data into the second feature extraction branch, perform feature extraction, and obtain multimodal features; S34: The multimodal features are spliced ​​and integrated according to the channel dimension to obtain the features of each modality at different scale levels.

[0008] Preferably, in step S4, the features of each modality at different scale levels are processed using a gradient-weighted class activation mapping method. During backpropagation, the contribution of each channel to the loss function is calculated to obtain the weight of each channel. Then, the weights of each channel are weighted to obtain the importance map of the features of each modality at each scale, including: S41: The loss function is obtained by employing structural consistency, multi-source information preservation, and perceptual loss; S42: Based on the loss function and the features of each modality at different scale levels, the weights of each channel are obtained through backpropagation calculation and global average pooling operation. The weights of each channel are used to characterize the overall contribution of the feature channel. S43: Based on the channel weights, the weighted feature map is obtained after channel weighting calculation; S44: A modified linear unit activation function is used to perform a nonlinear transformation on the weighted feature map to obtain an activated feature map. The nonlinear transformation is used to enhance the expressive power of the features and introduce nonlinearity. S45: Normalization is applied to adjust the numerical range of the activation feature map to obtain the importance map of each modality feature.

[0009] Preferably, step S5 utilizes the importance maps of features at each scale of each modality and the scale-adapted semantic attention map to calculate a dynamically weighted map of features at each scale of each modality. Finally, it performs a weighted multiplication with the features of multiple modalities at each scale to obtain a multimodal feature fusion map at different scales, including: S51: Based on the scale-adapted semantic attention map and the importance map of each modality at each scale, calculate the dynamic weighted map of each modality at each scale to obtain the dynamic weighted map. S52: An attention weight generation function is used to perform linear or nonlinear mapping on the dynamic weighted graph, and then Softmax normalization is performed to obtain the weight distribution. The Softmax normalization is limited to exponential normalization across channels or across modes, and the weight distribution satisfies that the sum of the weights is 1. S53: Based on the weight distribution, the dynamic weighted graph is multiplied with the importance graphs of each modality and each scale feature to obtain a multimodal feature fusion graph at different scales.

[0010] Preferably, step S6 involves inputting a multimodal feature fusion map of different scales into a decoding network, and obtaining a multimodal fused image through upsampling, normalization activation processing, and post-processing operations, including: S61: Construct a decoding network that shares a structure with the decoder of the improved IVMS-UNet, including transposed convolutional layers, normalization layers, ReLU activation functions, and channel attention modules; S62: Based on the decoding network, upsampling operation is performed on the multimodal feature fusion map at different scales to obtain the feature map after resolution restoration. The upsampling operation includes 4-level upsampling using transposed convolutional layers to gradually restore the fused features to the preset resolution. The transposed convolutional layer includes 4 layers of 2×2 transposed convolutions, with a stride of 2 and padding of 0 for each transposed convolution. S63: Based on the decoding network, the feature map after the resolution is restored is subjected to normalized activation processing to obtain normalized activation features. The normalized activation processing includes inputting the feature map after the resolution is restored into the normalization layer and the ReLU activation function after each transposed convolution. S64: Based on the decoding network, the channel weights of the normalized activation features are adaptively adjusted through the channel attention module to obtain the feature map after weight adjustment. The channel attention module is embedded in the decoding network. S65: Perform post-processing operations on the weighted feature map to obtain the final fused image. The post-processing operations include edge sharpening, color correction, feature channel compression, and activation processing. The edge sharpening includes edge sharpening using the Laplacian operator. The feature channel compression includes inputting a 1×1 convolutional layer to compress the feature channel number, resulting in a 3-channel feature map. The activation processing includes activating the 3-channel feature map using the Sigmoid activation function. The pixel value range of the fused image is a preset range.

[0011] On the other hand, a text-guided dynamic weight allocation multimodal fusion device is provided, which is applied to the text-guided dynamic weight allocation multimodal fusion method. The device includes: Image acquisition module: used to acquire multi-source, multi-modal input images, perform preprocessing, and obtain a standardized image set; Semantic similarity module: used to receive text queries and perform vectorization processing, extract features and calculate semantic similarity on a standardized image set to obtain a scale-adapted semantic attention map. The vectorization processing includes encoding the text query through a text encoder. Feature extraction network module: used to build a feature extraction network to extract features from each modality image in the standardized image set, and obtain the features of each modality at different scale levels; Loss function module: It is used to calculate the contribution of each channel to the loss function during backpropagation by using the gradient weighted activation mapping method to calculate the contribution of each channel to the loss function, obtain the weight of each channel, and then weight each channel to obtain the importance map of each feature at each scale of each modality. The fusion operation module is used to calculate the dynamic weighted map of the features at each scale of each modality using the importance map of each feature at each scale and the semantic attention map of the scale adaptation. Finally, it performs a weighted multiplication with the features of multiple modalities at each scale to obtain a multimodal feature fusion map at different scales. Decoding network module: It is used to input multimodal feature fusion maps of different scales into the decoding network, and obtain the multimodal fused image through upsampling operation, normalization activation processing and post-processing operation.

[0012] On the other hand, a text-guided dynamic weight allocation multimodal fusion device is provided, the text-guided dynamic weight allocation multimodal fusion device comprising: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement the method described in any of the above-described text-guided dynamic weight allocation multimodal fusion methods.

[0013] On the other hand, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores program code, which can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

[0014] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: Leveraging the cross-modal semantic matching capabilities of the Grounding DINO model, this method achieves precise binding between text queries and image semantics, accurately locating semantic regions related to the query in visible light images. Combining the inherent saliency of multi-source, multimodal image features, it dynamically calculates the fusion weights of each source image at different scales and in different regions. The dynamic weight allocation strategy adapts to different scenarios and needs, allowing for adjustments to the fusion target through text queries without retraining. This not only makes the fusion results more aligned with user needs, significantly improving information completeness and detail richness, but also lowers the operational threshold and demonstrates strong generalization ability. It can be widely applied in remote sensing, healthcare, security, and other fields, providing high-quality input data for downstream visual tasks to achieve semantically perceptive adaptive image fusion. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a text-guided dynamic weight allocation multimodal fusion method provided in an embodiment of the present invention; Figure 2 This is a structural diagram of a dynamic fusion method provided in an embodiment of the present invention; Figure 3 This is a block diagram of a text-guided dynamic weight allocation multimodal fusion device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a text-guided dynamic weight allocation multimodal fusion device provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0018] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0019] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0020] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0021] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0022] This invention provides a text-guided dynamic weight allocation multimodal fusion method, which can be implemented by a text-guided dynamic weight allocation multimodal fusion device, which can be a terminal or a server. Figure 1 The flowchart shown is for a text-guided dynamic weight allocation multimodal fusion method. The processing flow of this method may include the following steps:

[0023] Preferably, multi-source, multi-modal input images are acquired, preprocessed, and a standardized image set is obtained, including: Multi-source, multimodal input images are obtained by acquiring multi-source image data from different sensors; The multi-source, multimodal input images are uniformly scaled to a preset size using a scaling algorithm to obtain a uniform-size image. The scaling algorithm includes bilinear interpolation or nearest-neighbor interpolation methods, and the uniform-size image includes a single-channel image and / or a multispectral image. The uniform-sized image is channel-aligned to obtain a channel-aligned image. The channel alignment includes channel duplication of a single-channel image and selection of key channels in a multispectral image. Based on the noise content of the image, the channel-aligned image is denoised to obtain a denoised image. The denoising process includes noise suppression processing through Gaussian filtering, median filtering and / or adaptive Wiener filtering. The pixel values ​​of the denoised image are subjected to grayscale normalization processing and modal index construction to obtain a standardized image set. The grayscale normalization processing includes mapping the pixel values ​​to a preset range using the min-max normalization method.

[0024] In some embodiments, the multi-source multimodal input image includes visible light images and other modal images, such as infrared images, multispectral images, medical images, remote sensing images, security monitoring images, etc.; the specific preprocessing operations are as follows: all source images are uniformly scaled to a preset size (such as 512×512, 1024×1024) using bilinear interpolation or nearest neighbor interpolation methods to complete size normalization; single-channel images are copied, and key channels are selected for multispectral images to achieve channel alignment; pixel values ​​are mapped to the [0,1] range using the min-max normalization method to complete grayscale normalization; Gaussian filtering, median filtering, or adaptive Wiener filtering are used to suppress noise in noisy images.

[0025] In some embodiments, each modal image undergoes corresponding processing in sequence. For example, the visible light image retains the RGB three channels, the infrared image is converted to a single channel and normalized to the [0,1] range, and the multispectral image undergoes band registration and invalid band removal. The min-max normalization method is used to normalize the pixel values ​​of the visible light image, infrared image, and multispectral image to the [0,1] range. Median filtering (window size 3×3) is applied to the infrared image for noise suppression. The visible light image does not require additional filtering. The multispectral image undergoes selective light smoothing or principal component compression based on the noise level to reduce redundancy.

[0026] Preferably, the received text query is vectorized, and feature extraction and semantic similarity calculation are performed on a standardized image set to obtain a scale-adapted semantic attention map. The vectorization process includes encoding the text query using a text encoder, including: The system receives a text query and, based on the Transformer architecture, performs vectorization processing on the text query to generate a text query vector. The vectorization processing includes word segmentation, embedding, and multi-layer encoding. The text query vector has a dimension of 768. An image encoder is used to extract features from a standardized image set to obtain multi-scale feature maps. The image encoder is based on a ResNet or Swin Transformer architecture. The multi-scale feature maps contain low-level texture features and high-level semantic features. The high-level semantic features include a first-scale feature map, a second-scale feature map, a third-scale feature map, and / or a fourth-scale feature map. The resolution of the first-scale feature map is set to the initial resolution and no downsampling processing is required. The second-scale feature map is obtained by performing a 2×2 max pooling operation on the first-scale feature map, and the resolution of the second-scale feature map is 320×256. The third-scale feature map is obtained by performing a 4×4 max pooling operation on the second-scale feature map, and the resolution of the third-scale feature map is 160×128. The fourth-scale feature map is obtained by performing an 8×8 max pooling operation on the third-scale feature map, and the resolution of the fourth-scale feature map is 80×64. Based on the cross-attention mechanism, the semantic similarity between the text query vector and the high-level semantic features is calculated to locate the semantic region related to the text query, thereby obtaining a scale-adapted semantic attention map. The dimension of the scale-adapted semantic attention map is consistent with the dimension of the multi-scale feature map. The pixel values ​​in the semantic attention map are within a preset range, and the pixel values ​​are used to characterize the correlation between the corresponding region and the text query.

[0027] In some embodiments, such as Figure 2 As shown, the text encoder is based on the Transformer architecture and generates a 768-dimensional text query vector through word segmentation, embedding, and multi-layer encoding. The image encoder adopts the ResNet or SwinTransformer architecture to extract multi-scale feature maps containing low-level texture and high-level semantics. The cross-modal interaction submodule calculates the semantic similarity between the query vector and image features based on the cross-attention mechanism and outputs a semantic attention map with the same dimension as the image feature map. The pixel values ​​in the map range from [0,1], and the higher the value, the higher the correlation between the region and the text query.

[0028] In some embodiments, a pre-trained Grounding DINO model is loaded, its text encoder and image encoder parameters are frozen, and the cross-modal interaction submodule is fine-tuned only for multimodal image fusion scenarios; the core parameters of the word segmenter and Transformer encoder for the task are configured, ResNet-50 is used as the backbone network to extract image features, and by setting the relevant parameters of the attention mechanism and the number of cross-modal interactions, a semantic attention map of a specified dimension is finally output to adapt to the feature extraction requirements of the fusion scenario.

[0029] Preferably, a feature extraction network is constructed to extract features from each modality image in the standardized image set, obtaining features of each modality at different scale levels, including: A feature extraction network is established. The feature extraction architecture of the feature extraction network is based on the improved IVMS-UNet 4-level codec symmetric structure, including channel splitting and dual-branch parallel extraction architecture. The dual-branch parallel extraction architecture includes a first feature extraction branch and a second feature extraction branch. The decoder of the improved IVMS-UNet adopts a dual-channel codec symmetric architecture, which includes 4 levels of downsampling units and 4 levels of upsampling units. Based on a feature extraction network, a standardized image set is split according to the number of channels to obtain first channel data and second channel data. The splitting includes splitting according to the first and second halves of the number of channels. The first channel data is input into the first feature extraction branch, and the second channel data is input into the second feature extraction branch to perform feature extraction and obtain multimodal features; The multimodal features are spliced ​​and integrated according to the channel dimension to obtain the features of each modality at different scale levels.

[0030] It should be noted that, compared to the classic U-Net, the core differences and improvements of the IVMS-UNet dual-channel U-shaped feature extraction network are concentrated in the comprehensive upgrade of modality processing logic, feature extraction paradigm, and fusion strategy. Taking visible light and infrared images as examples, IVMS-UNet abandons the traditional U-shaped network's single-branch unified processing mode for input tensors. It innovatively adopts a channel splitting-dual-branch parallel extraction architecture, splitting the input tensor into two halves based on the number of channels and feeding them into the visible light and infrared feature extraction branches respectively. Both branches inherit and improve IVMS-UNet's 4-level encoding and decoding symmetric structure, which can specifically enhance the feature characteristics of different modalities (e.g., the visible light branch focuses on shallow details, and the infrared branch focuses on deep thermal target semantics), achieving decoupling of dual-modal feature extraction and avoiding the overlap of features from different modalities under a single branch. Simultaneously, it breaks through the limitation of traditional U-shaped networks relying solely on skip connections to achieve intra-modal scale fusion, adding a channel-dimensional cross-modal fusion strategy. This involves splicing and integrating the multi-scale feature maps extracted by the two branches along the channel dimension to form a new tensor that combines visible light detail texture and infrared semantic information. The final output is a multi-scale feature map that fuses dual-modal attributes, retaining the advantages of UNet's multi-scale feature extraction while effectively solving the problems of traditional U-shaped networks through the innovative design of "decoupled extraction + channel fusion." The core problem is that this type of network cannot adapt to visible light-infrared dual-modal data.

[0031] It should be noted that the improved IVMS-UNet network adopts a dual-channel encoding and decoding symmetrical architecture, which includes 4 levels of downsampling units (encoders) and 4 levels of upsampling units (decoders). The following description uses visible light images and infrared images as examples. The input tensor is split into two halves according to the number of channels. The first 1-3 channels are sent to the visible light feature extraction branch, and the last 4-6 channels are sent to the infrared feature extraction branch. The two branches are completely symmetrical.

[0032] Encoder feature extraction: Each downsampling unit contains two rounds of 3×3 convolution operations (stride 1, padding 1), followed by a batch normalization layer and a ReLU activation function, and finally downsampling is completed through a 2×2 max pooling layer (stride 2). After 4 levels of downsampling, the visible light branch outputs feature maps at 4 scales, with the number of channels being 64, 128, 256, and 512 respectively, while the infrared branch outputs feature maps at the same scale, with the same number of channels as the visible light branch.

[0033] Dual-channel feature fusion: At each scale level of the encoder, the feature maps of the visible light branch and the infrared branch are concatenated along the channel dimension to form a feature set that contains both dual-modal information. .

[0034] Semantic Attention Graph Downsampling to the feature set The corresponding resolution yields a scale-adapted semantic attention graph. The details are as follows: First-scale feature map With a resolution of 640×512, no downsampling is needed; simply set... ; Second-scale feature map The resolution is 320×256. Perform 2×2 max pooling to obtain ; 3rd Scale Feature Map The resolution is 160×128. Perform 4×4 max pooling to obtain ; 4th Scale Feature Map The resolution is 80×64. Perform 8×8 max pooling to obtain .

[0035] Preferably, the features of each modality at different scale levels are processed using a gradient-weighted class activation mapping method. During backpropagation, the contribution of each channel to the loss function is calculated to obtain the weight of each channel. Then, the weights of each channel are summed to obtain the importance map of the features of each modality at each scale, including: The loss function is obtained by employing structural consistency, multi-source information preservation, and perceptual loss. Based on the loss function and the features of each modality at different scale levels, the weights of each channel are obtained through backpropagation calculation and global average pooling operation. The weights of each channel are used to characterize the overall contribution of the feature channel. Based on the channel weights, a weighted feature map is obtained after channel weighting calculation. A modified linear unit activation function is used to perform a nonlinear transformation on the weighted feature map to obtain an activated feature map. The nonlinear transformation is used to enhance the expressive power of the features and introduce nonlinearity. By employing normalization processing, the numerical range of the activation feature map is adjusted to obtain the feature importance map for each modality.

[0036] In some embodiments, feature importance graphs Generated using the Gradient Weighted Class Activation Map (Grad-CAM) method, the first step is to define a loss function that measures the "fusion effect," serving as the objective for calculating the feature gradient (the gradient direction being the direction of "reducing loss and improving fusion quality"). The loss function is then defined as follows: .in, Used to measure the consistency of the fusion result with the source image in terms of structural information, ensuring that no details are lost; Used to improve the retention of multi-source information in the fusion results; To constrain high-level semantic consistency by perceiving loss, avoid the problem of "pixel alignment but semantic distortion" in fused images, and make the fusion result more in line with human visual perception habits; These are the weighting coefficients. They represent the s-scale source image features extracted by IVMS-UNet. Calculate its contribution to total loss gradient: . The c-th channel of the s-th scale feature map of the source image The pixel value of the location; The gradient represents the total loss with respect to the corresponding pixel value (the gradient magnitude reflects the sensitivity of the corresponding feature to changes in the fusion loss; a larger magnitude indicates a more significant impact of the feature on the fusion result); global average pooling is performed on the gradient map of each channel to obtain the "importance weight" of that channel—that is, the overall contribution of that channel to the fusion task. . and This represents the height and width of the feature map at scale s. This represents the weight of the c-th channel of the n-th modality image at scale s. A larger weight indicates that the features contained in that channel (such as edges, textures, and semantic regions) are more important for fusion. By using "channel weighted feature map + ReLU activation + normalization", feature importance maps of different modalities at each scale level are obtained. . Let be the number of channels in the feature map at scale s; the dynamic weighted map is calculated as follows: ,in This indicates a feature concatenation operation. This represents a weight generation function constructed based on the attention concept. Furthermore, the weighted graph is weighted along the modal dimension at the same scale *s*. Perform Softmax normalization to obtain the normalized dynamic weighted graph. ,in The softmax normalization function is... The weight of the nth modality at scale s represents the multimodal feature weighting fusion formula. Residual connections are introduced during the fusion process to enhance feature propagation.

[0037] Preferably, using the importance maps of features at each scale of each modality and the scale-adapted semantic attention map, a dynamic weighted map of features at each scale of each modality is calculated. Finally, this map is multiplied with the features of multiple modalities at each scale using weighted methods to obtain a multimodal feature fusion map at different scales, including: Based on the scale-adapted semantic attention map and the importance map of each modality at each scale, a dynamic weighted map of each modality at each scale is calculated to obtain the dynamic weighted map. An attention weight generation function is used to perform linear or nonlinear mapping on the dynamic weighted graph, and then Softmax normalization is performed to obtain the weight distribution. The Softmax normalization is limited to exponential normalization across channels or across modes, and the weight distribution satisfies that the sum of the weights is 1. Based on the weight distribution, the dynamic weighted graph is multiplied with the importance graphs of each modality at each scale to obtain a multimodal feature fusion graph at different scales.

[0038] In some embodiments, feature importance maps of different modalities at each scale level are obtained by channel weighted feature maps, ReLU activation, and normalization: Combining scale-adaptive semantic attention graphs Feature importance graphs of different modes at various scale levels Calculate the dynamic weighted graph The formula is as follows: In this embodiment, the settings are as follows: For attention weight generation function, Use the Softmax normalization function to ensure the sum of the weight values ​​is 1; [This is related to] the dynamically weighted graph. Feature maps of each modality at different scale levels By performing weighted multiplication, we obtain the fusion map of different modal features at this scale. : .

[0039] Preferably, multimodal feature fusion maps of different scales are input into the decoding network, and through upsampling, normalized activation processing, and post-processing, a multi-scale multimodal fused image is obtained, including: A decoding network is constructed, which shares a structure with the decoder of the improved IVMS-UNet, including transposed convolutional layers, normalization layers, ReLU activation functions, and channel attention modules; The decoding network performs upsampling operations on multimodal feature fusion maps at different scales to obtain feature maps with restored resolution. The upsampling operation includes four levels of upsampling using transposed convolutional layers to gradually restore the fused features to a preset resolution. The transposed convolutional layers include four 2×2 transposed convolutions, with a stride of 2 and padding of 0 for each transposed convolution. Based on the decoding network, the feature map after the resolution is restored is subjected to normalized activation processing to obtain normalized activation features. The normalized activation processing includes inputting the feature map after the resolution is restored into the normalization layer and the ReLU activation function after each transposed convolution. Based on the decoding network, the channel weights of the normalized activation features are adaptively adjusted through the channel attention module to obtain the weight-adjusted feature map. The channel attention module is embedded in the decoding network. The weighted feature map is post-processed to obtain the final fused image. The post-processing operations include edge sharpening, color correction, feature channel compression, and activation processing. The edge sharpening includes using the Laplacian operator for edge sharpening. The feature channel compression includes inputting a 1×1 convolutional layer to compress the feature channels to obtain a 3-channel feature map. The activation processing includes activating the 3-channel feature map using the Sigmoid activation function. The pixel value range of the fused image is a preset range.

[0040] In some embodiments, contrast-limited adaptive histogram equalization is employed to enhance image contrast and improve visual effects; Laplacian operator is used for edge sharpening to enhance image edge details; and color correction is performed to adjust the image color gamut to a natural range. The final image output is a fused image with pixel values ​​ranging from [0,1], obtained by compressing the feature channels to 3 channels using a 1×1 convolutional layer and applying a Sigmoid activation function. This completes the semantic-guided fusion of visible light and infrared images.

[0041] The above is an introduction to the method embodiments. The following describes the solution described in this application through device embodiments.

[0042] Figure 3 This is a block diagram of a text-guided dynamic weight allocation multimodal fusion apparatus according to an exemplary embodiment. The apparatus is used in a text-guided dynamic weight allocation multimodal fusion method. (Refer to...) Figure 3 The device includes an image acquisition module, a semantic similarity module, a feature extraction network module, a loss function module, a fusion operation module, and a decoding network module.

[0043] Image acquisition module: used to acquire multi-source, multi-modal input images, perform preprocessing, and obtain a standardized image set; Semantic similarity module: used to receive text queries and perform vectorization processing, extract features and calculate semantic similarity on a standardized image set to obtain a scale-adapted semantic attention map. The vectorization processing includes encoding the text query through a text encoder. Feature extraction network module: used to build a feature extraction network to extract features from each modality image in the standardized image set, and obtain the features of each modality at different scale levels; Loss function module: It is used to calculate the contribution of each channel to the loss function during backpropagation by using the gradient weighted activation mapping method to calculate the contribution of each channel to the loss function, obtain the weight of each channel, and then weight each channel to obtain the importance map of each feature at each scale of each modality. The fusion operation module is used to calculate the dynamic weighted map of the features at each scale of each modality using the importance map of each feature at each scale and the semantic attention map of the scale adaptation. Finally, it performs a weighted multiplication with the features of multiple modalities at each scale to obtain a multimodal feature fusion map at different scales. Decoding network module: It is used to input multimodal feature fusion maps of different scales into the decoding network, and obtain the multimodal fused image through upsampling operation, normalization activation processing and post-processing operation.

[0044] A text-guided dynamic weight allocation multimodal fusion device is provided, comprising: a processor; and a memory storing computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the method described in any of the above-described text-guided dynamic weight allocation multimodal fusion methods is implemented.

[0045] A computer-readable storage medium, characterized in that the computer-readable storage medium stores program code, the program code being invoked by a processor to execute the method as described in any one of claims 1 to 7.

[0046] Figure 4 This is a schematic diagram of the structure of a text-guided dynamic weight allocation multimodal fusion device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the text-guided dynamic weight allocation multimodal fusion device may include the above-mentioned... Figure 3 The illustrated text-guided dynamic weight allocation multimodal fusion device 410 may optionally include a first processor 2001.

[0047] Optionally, the text-guided dynamic weight allocation multimodal fusion device 410 may also include a memory 2002 and a transceiver 2003.

[0048] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0049] The following is combined with Figure 4 The components of the text-guided dynamic weight allocation multimodal fusion device 410 are described in detail below: The first processor 2001 is the control center of the text-guided dynamic weight allocation multimodal fusion device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0050] Optionally, the first processor 2001 can perform various functions of the text-guided dynamic weight allocation multimodal fusion device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0051] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0052] In a specific implementation, as one example, the text-guided dynamic weight allocation multimodal fusion device 410 may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0053] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0054] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the text-guided dynamic weight allocation multimodal fusion device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0055] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0056] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0057] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the text-guided dynamic weight allocation multimodal fusion device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0058] It should be noted that, ​ The structure of the text-guided dynamic weight allocation multimodal fusion device 410 shown in the figure does not constitute a limitation on the router. Actual knowledge structure recognition devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0059] Furthermore, the technical effects of the text-guided dynamic weight allocation multimodal fusion device 410 can be referred to the technical effects of the text-guided dynamic weight allocation multimodal fusion method described in the above method embodiments, and will not be repeated here.

[0060] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0061] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0062] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0063] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0064] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0065] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0066] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0067] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0068] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0069] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0070] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0071] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0072] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A text-guided dynamic weight allocation multimodal fusion method, characterized in that, The method includes: S1: Acquire multi-source, multi-modal input images, perform preprocessing, and obtain a standardized image set; S2: Receive text queries and perform vectorization processing, extract features and calculate semantic similarity on a standardized image set to obtain a scale-adapted semantic attention map. The vectorization processing includes encoding the text query through a text encoder. S3: Construct a feature extraction network to extract features from each modality image in the standardized image set, and obtain the features of each modality at different scale levels; S4: For the features of each modality at different scale levels, the gradient weighted class activation mapping method is used to calculate the contribution of each channel to the loss function during backpropagation, obtain the weight of each channel, and then weight each channel to obtain the importance map of each modality at each scale. S5: Using the importance maps of each modality and scale features and the scale-adapted semantic attention map, calculate the dynamic weighted map of each modality and scale features, and finally multiply it with the features of multiple modalities at each scale to obtain the multimodal feature fusion map of different scales. S6: Input the multimodal feature fusion map of different scales into the decoding network, and obtain the multimodal fused image through upsampling, normalization activation and post-processing operations.

2. The text-guided dynamic weight allocation multimodal fusion method according to claim 1, characterized in that, The S1 process acquires multi-source, multi-modal input images, performs preprocessing, and obtains a standardized image set, including: S11: Acquire multi-source image data from different sensors to obtain multi-source, multi-modal input images; S12: The multi-source multimodal input image is uniformly scaled to a preset size using a scaling algorithm to obtain a uniform-size image. The scaling algorithm includes bilinear interpolation or nearest neighbor interpolation methods. The uniform-size image includes a single-channel image and / or a multispectral image. S13: Perform channel alignment on the uniform-size image to obtain a channel-aligned image. The channel alignment includes channel duplication of a single-channel image and selection of key channels in a multispectral image. S14: Based on the noise content of the image, the channel-aligned image is denoised to obtain a denoised image. The denoising process includes noise suppression processing through Gaussian filtering, median filtering and / or adaptive Wiener filtering. S15: Perform grayscale normalization on the pixel values ​​of the denoised image and construct a modal index to obtain a standardized image set. The grayscale normalization process includes mapping the pixel values ​​to a preset range using the min-max normalization method.

3. The text-guided dynamic weight allocation multimodal fusion method according to claim 1, characterized in that, The received text query in S2 is vectorized, and features are extracted and semantic similarity is calculated from a standardized image set to obtain a scale-adapted semantic attention map. The vectorization process includes encoding the text query using a text encoder, including: S21: Receive a text query, and based on the Transformer architecture, perform vectorization processing on the text query to generate a text query vector. The vectorization processing includes word segmentation, embedding, and multi-layer encoding processing. The dimension of the text query vector is 768. S22: An image encoder is used to extract features from a standardized image set to obtain multi-scale feature maps. The image encoder is based on a ResNet or Swin Transformer architecture. The multi-scale feature maps contain low-level texture features and high-level semantic features. The high-level semantic features include a first-scale feature map, a second-scale feature map, a third-scale feature map, and / or a fourth-scale feature map. The resolution of the first-scale feature map is set to the initial resolution and no downsampling processing is required. The second-scale feature map is obtained by performing a 2×2 max pooling operation based on the first-scale feature map, and the resolution of the second-scale feature map is 320×256. The third-scale feature map is obtained by performing a 4×4 max pooling operation based on the second-scale feature map, and the resolution of the third-scale feature map is 160×128. The fourth-scale feature map is obtained by performing an 8×8 max pooling operation based on the third-scale feature map, and the resolution of the fourth-scale feature map is 80×64. S23: Based on the cross-attention mechanism, the semantic similarity between the text query vector and the high-level semantic features is calculated to locate the semantic region related to the text query, thereby obtaining a scale-adapted semantic attention map. The dimension of the scale-adapted semantic attention map is consistent with the dimension of the multi-scale feature map. The pixel value range in the semantic attention map is a preset range, and the pixel value is used to represent the correlation between the corresponding region and the text query.

4. The text-guided dynamic weight allocation multimodal fusion method according to claim 1, characterized in that, The feature extraction network constructed in S3 extracts features from each modality image in the standardized image set, obtaining features of each modality at different scale levels, including: S31: Establish a feature extraction network. The feature extraction architecture of the feature extraction network is based on the improved IVMS-UNet 4-level codec symmetric structure, including channel splitting and dual-branch parallel extraction architecture. The dual-branch parallel extraction architecture includes a first feature extraction branch and a second feature extraction branch. The decoder of the improved IVMS-UNet adopts a dual-channel codec symmetric architecture, which includes 4-level downsampling units and 4-level upsampling units. S32: Based on the feature extraction network, the standardized image set is split according to the number of channels to obtain the first channel data and the second channel data. The splitting includes splitting according to the first and second halves of the number of channels. S33: Input the first channel data into the first feature extraction branch, input the second channel data into the second feature extraction branch, perform feature extraction, and obtain multimodal features; S34: The multimodal features are spliced ​​and integrated according to the channel dimension to obtain the features of each modality at different scale levels.

5. The text-guided dynamic weight allocation multimodal fusion method according to claim 1, characterized in that, In step S4, the features of each modality at different scale levels are processed using a gradient-weighted class activation mapping method. During backpropagation, the contribution of each channel to the loss function is calculated to obtain the weight of each channel. Then, the weights of each channel are summed to obtain the importance map of the features of each modality at each scale, including: S41: The loss function is obtained by employing structural consistency, multi-source information preservation, and perceptual loss; S42: Based on the loss function and the features of each modality at different scale levels, the weights of each channel are obtained through backpropagation calculation and global average pooling operation. The weights of each channel are used to characterize the overall contribution of the feature channel. S43: Based on the channel weights, the weighted feature map is obtained after channel weighting calculation; S44: A modified linear unit activation function is used to perform a nonlinear transformation on the weighted feature map to obtain an activated feature map. The nonlinear transformation is used to enhance the expressive power of the features and introduce nonlinearity. S45: Normalization is applied to adjust the numerical range of the activation feature map to obtain the importance map of each modality feature.

6. The text-guided dynamic weight allocation multimodal fusion method according to claim 1, characterized in that, S5 utilizes the importance maps of features at each scale of each modality and the scale-adapted semantic attention map to calculate a dynamically weighted map of features at each scale of each modality. Finally, it performs a weighted multiplication with the features of multiple modalities at each scale to obtain a multimodal feature fusion map at different scales, including: S51: Based on the scale-adapted semantic attention map and the importance map of each modality at each scale, calculate the dynamic weighted map of each modality at each scale to obtain the dynamic weighted map. S52: An attention weight generation function is used to perform linear or nonlinear mapping on the dynamic weighted graph, and then Softmax normalization is performed to obtain the weight distribution. The Softmax normalization is limited to exponential normalization across channels or across modes, and the weight distribution satisfies that the sum of the weights is 1. S53: Based on the weight distribution, the dynamic weighted graph is multiplied with the importance graphs of each modality and each scale feature to obtain a multimodal feature fusion graph at different scales.

7. The text-guided dynamic weight allocation multimodal fusion method according to claim 1, characterized in that, The S6 inputs the multimodal feature fusion map of different scales into the decoding network, and obtains the multimodal fused image through upsampling, normalization activation processing, and post-processing operations, including: S61: Construct a decoding network that shares a structure with the decoder of the improved IVMS-UNet, including transposed convolutional layers, normalization layers, ReLU activation functions, and channel attention modules; S62: Based on the decoding network, upsampling operation is performed on the multimodal feature fusion map at different scales to obtain the feature map after resolution restoration. The upsampling operation includes 4-level upsampling using transposed convolutional layers to gradually restore the fused features to the preset resolution. The transposed convolutional layer includes 4 layers of 2×2 transposed convolutions, with a stride of 2 and padding of 0 for each transposed convolution. S63: Based on the decoding network, the feature map after the resolution is restored is subjected to normalized activation processing to obtain normalized activation features. The normalized activation processing includes inputting the feature map after the resolution is restored into the normalization layer and the ReLU activation function after each transposed convolution. S64: Based on the decoding network, the channel weights of the normalized activation features are adaptively adjusted through the channel attention module to obtain the feature map after weight adjustment. The channel attention module is embedded in the decoding network. S65: Perform post-processing operations on the weighted feature map to obtain the final fused image. The post-processing operations include edge sharpening, color correction, feature channel compression, and activation processing. The edge sharpening includes edge sharpening using the Laplacian operator. The feature channel compression includes inputting a 1×1 convolutional layer to compress the feature channel number, resulting in a 3-channel feature map. The activation processing includes activating the 3-channel feature map using the Sigmoid activation function. The pixel value range of the fused image is a preset range.

8. A text-guided dynamic weight allocation multimodal fusion device, wherein the text-guided dynamic weight allocation multimodal fusion device is used to implement the text-guided dynamic weight allocation multimodal fusion method as described in any one of claims 1-7, characterized in that, The device includes: Image acquisition module: used to acquire multi-source, multi-modal input images, perform preprocessing, and obtain a standardized image set; Semantic similarity module: used to receive text queries and perform vectorization processing, extract features and calculate semantic similarity on a standardized image set to obtain a scale-adapted semantic attention map. The vectorization processing includes encoding the text query through a text encoder. Feature extraction network module: used to build a feature extraction network to extract features from each modality image in the standardized image set, and obtain the features of each modality at different scale levels; Loss function module: It is used to calculate the contribution of each channel to the loss function during backpropagation by using the gradient weighted activation mapping method to calculate the contribution of each channel to the loss function, obtain the weight of each channel, and then weight each channel to obtain the importance map of each feature at each scale of each modality. The fusion operation module is used to calculate the dynamic weighted map of the features at each scale of each modality using the importance map of each feature at each scale and the semantic attention map of the scale adaptation. Finally, it performs a weighted multiplication with the features of multiple modalities at each scale to obtain a multimodal feature fusion map at different scales. Decoding network module: It is used to input multimodal feature fusion maps of different scales into the decoding network, and obtain the multimodal fused image through upsampling operation, normalization activation processing and post-processing operation.

9. A text-guided dynamic weight allocation multimodal fusion device, characterized in that, The text-guided dynamic weight allocation multimodal fusion processor; a memory storing computer-readable instructions, which, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images