Cross-modal feature fusion image generation method, system and device based on semantic guidance and medium
By combining global semantic vectors with a diffusion model through a U-Net structure, text semantics and image features are deeply adaptively fused across multiple dimensions, improving the semantic accuracy and generation quality of cross-modal image generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANDONG INSPUR SCI RES INST CO LTD
- Filing Date
- 2025-11-27
- Publication Date
- 2026-04-17
AI Technical Summary
In existing cross-modal image generation technologies, the interaction between text semantics and image features is mostly shallow weighting, which fails to achieve adaptive fusion of global semantic vectors and image vectors in multiple dimensions, resulting in blurred details and poor semantic consistency in the generated images.
By combining multi-dimensional descriptive vectors of text data obtained through a pre-trained language model into a global semantic vector, and combining it with the U-Net structure of the cross-attention module and diffusion model, a deep and adaptive fusion of global semantic vectors and image vectors in multiple dimensions is achieved, generating semantically guided feature maps, which are then injected into the generation process of the diffusion model.
It improves the semantic accuracy and overall quality of generated images, ensuring a high degree of consistency and visual realism between generated images and text descriptions in terms of content, style, and other dimensions, and solves the problem of image content deviation or semantic ambiguity caused by insufficient fusion.
Smart Images
Figure CN121884050A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of RTMP one-way proxy technology, and in particular to a semantically guided cross-modal feature fusion image generation method, system, device and medium. Background Technology
[0002] In recent years, with the rapid development of artificial intelligence technology, cross-modal image generation technology has become a research hotspot in the fields of computer vision and natural language processing. Traditional methods mainly rely on models such as generative adversarial networks (GANs) or variational autoencoders (VAEs) to achieve cross-modal fusion by directly concatenating text features with image features. However, such methods suffer from problems such as inaccurate semantic alignment and limited feature interaction capabilities, resulting in blurry details and poor semantic consistency in the generated images. For example, when the text description involves complex scenes or multiple object relationships, existing technologies struggle to accurately capture the correspondence between semantic details and spatial structures, limiting the realism and diversity of the generated images. Furthermore, traditional diffusion models lack fine-grained guidance on text semantics during the generation process, and it is difficult to achieve deep collaboration of cross-modal features through simple conditional injection alone, affecting the generation quality and controllability.
[0003] To overcome these shortcomings, researchers have attempted to introduce attention mechanisms to enhance cross-modal interaction, such as linking text and image features through cross-attention modules. However, existing methods typically employ unidirectional attention computation, failing to fully exploit the dynamic guiding role of semantic information in the image generation process. Specifically, in the feature fusion stage, the interaction between text semantics and image features is mostly shallow weighting, failing to achieve adaptive fusion of global semantic vectors and image vectors across multiple dimensions. Summary of the Invention
[0004] This application provides a semantically guided cross-modal feature fusion image generation method, system, device, and medium to solve the problem that in the feature fusion stage of existing solutions, the interaction between text semantics and image features is mostly shallow weighting, and the adaptive fusion of global semantic vectors and image vectors in multiple dimensions is not achieved.
[0005] Firstly, this application provides a semantically guided cross-modal feature fusion image generation method, the method comprising: Obtain text data and corresponding image data; use a pre-trained language model to obtain description vectors of several preset dimensions corresponding to the text data; combine the description vectors into a global semantic vector; obtain the global image vector of the image data. Using global semantic vectors, calculate K and V values across attention modules; using global image vectors, calculate Q values across attention modules; perform a dot product operation between Q and K values, and normalize using the softmax function to obtain the attention weight matrix; use the attention weight matrix to perform a weighted summation of V values to output semantically guided feature maps. The semantically guided feature map is injected into the U-Net structure of the diffusion model to obtain the encoder, intermediate layer and decoder adjusted by the semantically guided feature map; the actual semantically guided text data is obtained, and the U-Net structure of the diffusion model outputs an image that conforms to the actual semantically guided text data.
[0006] In one implementation of this application, before using a pre-trained language model to obtain description vectors of several preset dimensions corresponding to the text data, and combining the description vectors into a global semantic vector, the method further includes: Using the sample dataset of the current preset dimension, obtain the fully connected branch network trained in the current preset dimension; where the sample dataset consists of sample text and labels of the current preset dimension; Configure pre-trained fully connected branch networks corresponding to each preset dimension at the input end of the pre-trained language model; among them, the preset dimensions include at least: topic, style, emotion and detail dimensions; Using a pre-trained language model, description vectors of several preset dimensions are obtained corresponding to the text data. These description vectors are then combined into a global semantic vector, specifically including: Text data is input into a pre-trained language model, along with pre-trained fully connected branch networks corresponding to each preset dimension. Each branch network outputs a d-dimensional description vector. Each descriptive vector is normalized and then weighted and concatenated to form a global semantic vector.
[0007] In one implementation of this application, the K and V values across attention modules are calculated using global semantic vectors, specifically including: pass: , Using the global semantic vector F, calculate the K and V values; in, , This represents the preset weight matrix.
[0008] In one implementation of this application, the Q-value across attention modules is calculated using a global image vector, specifically including: pass The Q value is calculated using the global image vector Z; in, This represents the preset weight matrix.
[0009] In one implementation of this application, the Q-value and K-value are multiplied by a dot product, and the attention weight matrix is obtained by normalization using the softmax function, specifically including: Through the formula: Calculate the attention weight matrix A.
[0010] In one implementation of this application, semantically guided feature maps are injected into the U-Net structure of the diffusion model to obtain an encoder, intermediate layer, and decoder adjusted by the semantically guided feature maps, specifically including: The semantically guided feature map is projected onto the channel dimension that matches the feature map of each layer of U-Net, and then fused with the downsampled feature map in the encoder using an element-wise addition method to obtain fused data; In the intermediate layer, K-keys and V-values are provided from the semantic feature map. The fused data is used as the Q-value to construct a cross-attention computation module. Then, the initial feature map is obtained by utilizing the cross-attention computation module. The decoder employs adaptive instance normalization or feature modulation to adjust the initial feature map using semantic information from the semantically guided feature map as scaling and offset parameters, thereby obtaining the final output image.
[0011] Secondly, this application provides a semantically guided cross-modal feature fusion image generation system, the system comprising: The acquisition module is used to acquire text data and corresponding image data. Using a pre-trained language model, it acquires description vectors of several preset dimensions corresponding to the text data and combines the description vectors into a global semantic vector; it also acquires the global image vector of the image data. The mapping module is used to calculate the K and V values across attention modules using global semantic vectors; calculate the Q value across attention modules using global image vectors; perform a dot product operation between the Q value and the K value, and normalize it using the softmax function to obtain the attention weight matrix; use the attention weight matrix to perform a weighted summation of the V value to output the semantically guided feature map. The output module is used to inject semantically guided feature maps into the U-Net structure of the diffusion model to obtain encoders, intermediate layers, and decoders adjusted by semantically guided feature maps; it also obtains actual semantically guided text data and outputs images that conform to the actual semantically guided text data through the U-Net structure of the diffusion model.
[0012] In one implementation of this application, the output module includes an output unit. The semantically guided feature map is projected onto the channel dimension that matches the feature map of each layer of U-Net. The encoder then fuses the feature map with the downsampled feature map using an element-wise addition method to obtain fused data. In the intermediate layer, K-keys and V-values are provided from the semantic feature map. The fused data is used as the Q-value to construct a cross-attention computation module. Then, the initial feature map is obtained by utilizing the cross-attention computation module. The decoder employs adaptive instance normalization or feature modulation to adjust the initial feature map using semantic information from the semantically guided feature map as scaling and offset parameters, thereby obtaining the final output image.
[0013] Thirdly, this application provides a semantically guided cross-modal feature fusion image generation device, the device comprising: processor; And a memory containing executable code, which, when executed, causes the processor to perform a semantically guided cross-modal feature fusion image generation method as described above.
[0014] Fourthly, this application provides a non-volatile computer storage medium storing computer instructions thereon, which, when executed, implement a semantically guided cross-modal feature fusion image generation method as described above.
[0015] As can be seen from the above technical solutions, this application has the following advantages: First, this application achieves deep and adaptive fusion of text semantics and image features across multiple dimensions, overcoming the limitations of shallow weighted fusion in existing technologies. Specifically, it extracts several pre-trained language models from text data, creating description vectors of several preset dimensions, and combines them into a global semantic vector, while simultaneously acquiring the global image vector of the image. This design ensures that text semantic information is fully deconstructed into multi-dimensional features, providing a rich and structured semantic foundation for deep fusion. Furthermore, the scheme utilizes the global semantic vector to calculate key values (K and V values) in the cross-attention mechanism and uses the global image vector to calculate query values (Q values). This mechanism enables image features to proactively and selectively retrieve and extract relevant information from multi-dimensional text semantics, generate an attention weight matrix through dot product operations and Softmax normalization, and finally perform a weighted summation of the semantic vectors to output a semantically guided feature map. The entire process is essentially an intensive, non-linear interactive computation between the global semantic vector and the global image vector in the feature space. It realizes the "inside-out" deep guidance of semantic information on the image feature generation process, rather than simple feature splicing or shallow weighting, thereby ensuring that the generated image maintains a high degree of intrinsic consistency with the text semantics in multiple dimensions such as content and style.
[0016] Second, this application improves the semantic accuracy and overall quality of the generated images by seamlessly injecting the deeply fused semantic feature maps into the core generation structure of the diffusion model. The semantically guided feature maps output from the aforementioned steps are injected into the U-Net structure of the diffusion model to adjust its encoder, intermediate layers, and decoder. As a key component of the diffusion model for denoising and image generation, the features at each level of the U-Net directly participate in the construction of the final image. Through multi-level (encoder, intermediate layer, decoder) semantic feature injection along this core path, the global semantic information of the text can continuously and coherently influence the entire image generation process, from coarse to fine, from abstract to concrete. This "end-to-end" semantic guidance mechanism ensures that during model inference, each generation step of the diffusion model is subject to enhanced semantic constraints and guidance for the actual semantically guided text data input. The final output image not only closely matches the core content of the text description but also accurately grasps its implicit style, composition, and detail requirements, improving the semantic fidelity and visual realism of the generated result relative to the guiding text and solving the problem of image content deviation or semantic ambiguity caused by insufficient fusion. Attached Figure Description To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 This is a flowchart of a semantically guided cross-modal feature fusion image generation method provided in an embodiment of this application.
[0018] Figure 2 This is a schematic diagram of the internal structure of a semantically guided cross-modal feature fusion image generation system provided in an embodiment of this application.
[0019] Figure 3 This is a schematic diagram of the internal structure of a semantically guided cross-modal feature fusion image generation device provided in an embodiment of this application. Detailed Implementation
[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0021] Those skilled in the art should understand that the embodiments described below are merely preferred embodiments of this disclosure and do not imply that this disclosure can only be implemented through these preferred embodiments. These preferred embodiments are merely used to explain the technical principles of this disclosure and are not intended to limit the scope of protection of this disclosure. Based on the preferred embodiments provided by this disclosure, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of this disclosure.
[0022] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0023] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0024] The embodiment provides a semantically guided cross-modal feature fusion image generation method, such as Figure 1 As shown in the embodiments of this application, the method mainly includes the following steps: Step 110: Obtain text data and corresponding image data. Using a pre-trained language model, obtain description vectors of several preset dimensions corresponding to the text data, and combine the description vectors into a global semantic vector; obtain the global image vector of the image data.
[0025] It should be noted that in step 110, the global image vector of the image data is obtained specifically through a pre-trained convolutional neural network. This convolutional neural network adopts the ResNet-50 architecture and is pre-trained on the ImageNet dataset. After adjusting the input image to a resolution of 224×224 pixels, feature extraction is performed through the convolutional and pooling layers of the convolutional neural network. Finally, a 2048-dimensional feature vector is obtained from before the fully connected layers as the global image vector. For the alignment processing of text data and image data, the global semantic vector and the global image vector are mapped to the same feature space. The dimensions of the two are unified to 512 dimensions through linear transformation, and their semantic relevance is calculated using cosine similarity.
[0026] In some embodiments, before using a pre-trained language model to obtain description vectors of several preset dimensions corresponding to the text data, and combining the description vectors into a global semantic vector, the method further includes: Using the sample dataset of the current preset dimension, obtain the fully connected branch network trained in the current preset dimension; where the sample dataset consists of sample text and labels of the current preset dimension; Configure pre-trained fully connected branch networks corresponding to each preset dimension at the input end of the pre-trained language model; among them, the preset dimensions include at least: topic, style, emotion and detail dimensions; Using a pre-trained language model, description vectors of several preset dimensions are obtained corresponding to the text data. These description vectors are then combined into a global semantic vector, specifically including: Text data is input into a pre-trained language model, along with pre-trained fully connected branch networks corresponding to each preset dimension. Each branch network outputs a d-dimensional description vector. Each descriptive vector is normalized and then weighted and concatenated to form a global semantic vector.
[0027] It should be further explained that the weighted concatenation method for the descriptive vectors is implemented as follows: The descriptive vectors for the four dimensions of theme, style, emotion, and detail are assigned weight coefficients of 0.4, 0.3, 0.2, and 0.1, respectively. Before weighted summation, each descriptive vector is first subjected to L2 normalization to eliminate differences in dimensions. Then, the weighted descriptive vectors are concatenated along the feature dimensions to form the final global semantic vector. The total dimension of this global semantic vector is 4d, where d represents the original dimension of each descriptive vector.
[0028] Furthermore, the pre-trained language model uses the BERT-base architecture with a hidden layer dimension of 768. Each fully connected branch network contains two fully connected layers: the first layer has a dimension of 768×512 and uses the ReLU activation function; the second layer has a dimension of 512×d, where d is set according to the specific dimensional characteristics: topic dimension d=256, style dimension d=128, sentiment dimension d=64, and detail dimension d=192. The branch network is trained using the cross-entropy loss function, fine-tuned on the labeled dataset of the corresponding dimension, with a training period of 50 epochs and a batch size of 32.
[0029] Step 120: Calculate the K and V values across attention modules using the global semantic vector; calculate the Q value across attention modules using the global image vector; perform a dot product operation between the Q and K values, and normalize them using the softmax function to obtain the attention weight matrix; use the attention weight matrix to perform a weighted summation of the V values to output the semantically guided feature map.
[0030] It's worth noting that the Cross-Attention Module is an attention mechanism widely used in deep learning, especially in the Transformer architecture and its variants. Its core function is to enable information exchange and association between two different sequences, rather than focusing solely on relationships within the sequence itself, as is the case with Self-Attention. The computation process of Cross-Attention involves three core components: Query, Key, and Value.
[0031] Specifically, the K and V values across attention modules are calculated using global semantic vectors, including: pass: , Using the global semantic vector F, calculate the K and V values; in, , This represents the preset weight matrix.
[0032] The calculation of the Q-value across attention modules specifically includes: pass The Q value is calculated using the global image vector Z; in, This represents the preset weight matrix.
[0033] The attention weight matrix is obtained by performing a dot product operation between the Q-value and the K-value, and then normalizing it using the softmax function. Specifically, it includes: Through the formula: Calculate the attention weight matrix A.
[0034] It should be noted that in step 120, the specific implementation of the cross-attention module is as follows: The key vector K and value vector V are calculated using the global semantic vector F, where F is a 512-dimensional vector. This is achieved through two independent linear transformation layers, using weight matrices W_K and W_V, each with a dimension of 512×512. Simultaneously, the query vector Q is calculated using the global image vector Z, which is also a 512-dimensional vector. This is then linearly transformed using the weight matrix W_Q, which also has a dimension of 512×512. These three weight matrices are initialized with a Xavier uniform distribution during model initialization.
[0035] When performing a dot product operation on Q and K, Q and K are first aligned in dimensions, and the dot product result is divided by a scaling factor. ( =512) to stabilize the gradient. Then, the attention score is normalized using the softmax function to form the attention weight matrix A. The softmax function normalizes along the last dimension to ensure that the sum of all weight values is 1.
[0036] When the attention weight matrix A is used to perform a weighted summation on V, matrix multiplication is employed to output a semantically guided feature map. This feature map maintains the same dimensionality as V, being 512-dimensional. During computation, 32-bit floating-point precision is used, and the numerical range of the attention weights is constrained to the interval [0,1]. The final output feature map will serve as the input for subsequent image generation networks.
[0037] Step 130: Inject semantically guided feature maps into the U-Net structure of the diffusion model to obtain encoders, intermediate layers, and decoders adjusted by semantically guided feature maps; obtain actual semantically guided text data, and output images that conform to the actual semantically guided text data through the U-Net structure of the diffusion model.
[0038] It should be noted that the U-Net structure of the diffusion model is an important architecture for image generation tasks. Its core design is based on an encoder-decoder structure, with feature connections in the intermediate layers.
[0039] In some embodiments, semantically guided feature maps are injected into the U-Net structure of the diffusion model to obtain encoders, intermediate layers, and decoders adjusted by semantically guided feature maps, specifically including: The semantically guided feature map is projected onto the channel dimension that matches the feature map of each layer of U-Net, and then fused with the downsampled feature map in the encoder using an element-wise addition method to obtain fused data; In the intermediate layer, K-keys and V-values are provided from the semantic feature map. The fused data is used as the Q-value to construct a cross-attention computation module. Then, the initial feature map is obtained by utilizing the cross-attention computation module. The decoder employs adaptive instance normalization or feature modulation to adjust the initial feature map using semantic information from the semantically guided feature map as scaling and offset parameters, thereby obtaining the final output image.
[0040] It should be further explained that, in the encoder stage, the semantically guided feature map is first projected through a 1×1 convolutional layer to a channel dimension that matches the feature maps of each level of U-Net. Specifically, for a downsampled feature map with an input size of H×W×C, the semantic feature map is adjusted to the same spatial size H×W, and the number of channels is unified to C through a convolutional kernel. During the fusion process, the adjusted semantic feature map is fused with the corresponding level of U-Net downsampled feature map using an element-wise addition method. This fusion method ensures that semantic information is integrated early in feature extraction while maintaining the integrity of the original image features. The fused feature map contains both low-level texture and structural information and high-level semantic guidance, providing a rich feature representation foundation for subsequent processing.
[0041] In the intermediate processing stage, a cross-attention computation module was constructed to achieve deep semantic fusion. Specifically, key (K) and value (V) vectors were extracted from the semantic feature map, and the fused data output by the encoder was used as the query (Q) vector. The attention weights were calculated using the standard scaled dot product attention mechanism. Through this mechanism, the model can reweight the fused features according to the importance of semantic features, highlighting feature regions relevant to semantic description and suppressing irrelevant feature responses. The initial feature map output by this module maintains spatial consistency while strengthening the feature representation related to semantic guidance.
[0042] In the decoder stage, the initial feature map is finely adjusted using either Adaptive Instance Normalization (AdaIN) or feature modulation. For the AdaIN method, the semantic information in the semantically guided feature map is transformed into scaling parameter γ and offset parameter β. By normalizing the initial feature map, the final output image is obtained.
[0043] In addition, this application Figure 2 This application provides an embodiment of a semantically guided cross-modal feature fusion image generation system. For example... Figure 2 As shown in the embodiments of this application, the system mainly includes: The acquisition module 210 is used to acquire text data and corresponding image data. It uses a pre-trained language model to acquire description vectors of several preset dimensions corresponding to the text data and combines the description vectors into a global semantic vector; it also acquires the global image vector of the image data.
[0044] The mapping module 220 is used to calculate the K and V values across attention modules using global semantic vectors; calculate the Q value across attention modules using global image vectors; perform a dot product operation between the Q value and the K value, and normalize it using the softmax function to obtain the attention weight matrix; use the attention weight matrix to perform a weighted summation of the V value to output the semantically guided feature map.
[0045] The output module 230 is used to inject semantically guided feature maps into the U-Net structure of the diffusion model to obtain encoders, intermediate layers and decoders adjusted by semantically guided feature maps; to obtain actual semantically guided text data, and to output images that conform to the actual semantically guided text data through the U-Net structure of the diffusion model.
[0046] Output module 230 includes an output unit, The semantically guided feature map is projected onto the channel dimension that matches the feature map of each layer of U-Net. The encoder then fuses the feature map with the downsampled feature map using an element-wise addition method to obtain fused data. In the intermediate layer, K-keys and V-values are provided from the semantic feature map. The fused data is used as the Q-value to construct a cross-attention computation module. Then, the initial feature map is obtained by utilizing the cross-attention computation module. The decoder employs adaptive instance normalization or feature modulation to adjust the initial feature map using semantic information from the semantically guided feature map as scaling and offset parameters, thereby obtaining the final output image.
[0047] The above are method embodiments of this application. Based on the same inventive concept, this application also provides a semantically guided cross-modal feature fusion image generation device. Figure 3 As shown, the device includes: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform a semantically guided cross-modal feature fusion image generation method as described in the above embodiments.
[0048] Specifically, the server acquires text data and corresponding image data. Using a pre-trained language model, it obtains description vectors of several preset dimensions corresponding to the text data and combines these description vectors into a global semantic vector. It also acquires the global image vector of the image data. Using the global semantic vector, it calculates the K and V values across attention modules. Using the global image vector, it calculates the Q value across attention modules. The Q value and K value are multiplied by a dot product and normalized using a softmax function to obtain the attention weight matrix. The attention weight matrix is used to perform a weighted summation of the V values to output a semantically guided feature map. This semantically guided feature map is injected into the U-Net structure of the diffusion model to obtain the encoder, intermediate layers, and decoder adjusted by the semantically guided feature map. Finally, it acquires the actual semantically guided text data and outputs an image that matches the actual semantically guided text data through the U-Net structure of the diffusion model.
[0049] In addition, embodiments of this application also provide a non-volatile computer storage medium storing executable instructions, which, when executed, implement the semantically guided cross-modal feature fusion image generation method described above.
[0050] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A semantic guidance based cross-modal feature fusion image generation method, characterized in that, The method includes: Obtain text data and corresponding image data; use a pre-trained language model to obtain description vectors of several preset dimensions corresponding to the text data; combine the description vectors into a global semantic vector; obtain the global image vector of the image data. Using global semantic vectors, calculate K and V values across attention modules; using global image vectors, calculate Q values across attention modules; perform a dot product operation between Q and K values, and normalize using the softmax function to obtain the attention weight matrix; use the attention weight matrix to perform a weighted summation of V values to output semantically guided feature maps. The semantically guided feature map is injected into the U-Net structure of the diffusion model to obtain the encoder, intermediate layer and decoder adjusted by the semantically guided feature map; the actual semantically guided text data is obtained, and the U-Net structure of the diffusion model outputs an image that conforms to the actual semantically guided text data.
2. The semantic guidance based cross-modal feature fusion image generation method according to claim 1, characterized in that, Before using a pre-trained language model to obtain description vectors of several preset dimensions corresponding to the text data, and combining the description vectors into a global semantic vector, the method further includes: Using the sample dataset of the current preset dimension, obtain the fully connected branch network trained in the current preset dimension; where the sample dataset consists of sample text and labels of the current preset dimension; Configure pre-trained fully connected branch networks corresponding to each preset dimension at the input end of the pre-trained language model; among them, the preset dimensions include at least: topic, style, emotion and detail dimensions; Using a pre-trained language model, description vectors of several preset dimensions are obtained corresponding to the text data. These description vectors are then combined into a global semantic vector, specifically including: Text data is input into a pre-trained language model, along with pre-trained fully connected branch networks corresponding to each preset dimension. Each branch network outputs a d-dimensional description vector. Each descriptive vector is normalized and then weighted and concatenated to form a global semantic vector. 3.The semantic guidance based cross-modal feature fusion image generation method of claim 1, characterized in that, Using global semantic vectors, calculate the K and V values across attention modules, specifically including: By: , , using the global semantic vector F, calculate the K value and V value; wherein , denotes a preset weight matrix.
4. The semantic guidance based cross-modal feature fusion image generation method according to claim 1, characterized in that, Calculate the Q-value across attention modules using global image vectors, specifically including: By , the Q value is calculated using the global image vector Z; wherein, denotes a preset weight matrix.
5. The semantic guidance based cross-modal feature fusion image generation method according to claim 1, characterized in that, The attention weight matrix is obtained by performing a dot product operation between the Q-value and the K-value, and then normalizing it using the softmax function. Specifically, it includes: The attention weight matrix A is calculated by the formula: .
6. The semantically guided cross-modal feature fusion image generation method according to claim 1, characterized in that, Semantically guided feature maps are injected into the U-Net structure of the diffusion model to obtain encoders, intermediate layers, and decoders adjusted by semantically guided feature maps, specifically including: The semantically guided feature map is projected onto the channel dimension that matches the feature map of each layer of U-Net, and then fused with the downsampled feature map in the encoder using an element-wise addition method to obtain fused data; In the intermediate layer, K-keys and V-values are provided from the semantic feature map. The fused data is used as the Q-value to construct a cross-attention computation module. Then, the initial feature map is obtained by utilizing the cross-attention computation module. The decoder employs adaptive instance normalization or feature modulation to adjust the initial feature map using semantic information from the semantically guided feature map as scaling and offset parameters, thereby obtaining the final output image.
7. A semantically guided cross-modal feature fusion image generation system, characterized in that, The system includes: The acquisition module is used to acquire text data and corresponding image data. It uses a pre-trained language model to acquire description vectors of several preset dimensions corresponding to the text data and combines the description vectors into a global semantic vector; it also acquires the global image vector of the image data. The mapping module is used to calculate the K and V values across attention modules using global semantic vectors; calculate the Q value across attention modules using global image vectors; perform a dot product operation between the Q value and the K value, and normalize it using the softmax function to obtain the attention weight matrix; use the attention weight matrix to perform a weighted summation of the V value to output the semantically guided feature map. The output module is used to inject semantically guided feature maps into the U-Net structure of the diffusion model to obtain encoders, intermediate layers, and decoders adjusted by semantically guided feature maps; it also acquires actual semantically guided text data and outputs images that conform to the actual semantically guided text data through the U-Net structure of the diffusion model.
8. The semantically guided cross-modal feature fusion image generation system according to claim 7, characterized in that, The output module includes output units. The semantically guided feature map is projected onto the channel dimension that matches the feature map of each layer of U-Net. The encoder then fuses the feature map with the downsampled feature map using an element-wise addition method to obtain fused data. In the intermediate layer, K-keys and V-values are provided from the semantic feature map. The fused data is used as the Q-value to construct a cross-attention computation module. Then, the initial feature map is obtained by utilizing the cross-attention computation module. The decoder employs adaptive instance normalization or feature modulation to adjust the initial feature map using semantic information from the semantically guided feature map as scaling and offset parameters, thereby obtaining the final output image.
9. A semantically guided cross-modal feature fusion image generation device, characterized in that, The device includes: processor; And a memory having executable code stored thereon, which, when executed, causes the processor to perform a semantically guided cross-modal feature fusion image generation method as described in any one of claims 1-6.
10. A non-volatile computer storage medium, characterized in that, It stores computer instructions, which, when executed, implement a semantically guided cross-modal feature fusion image generation method as described in any one of claims 1-6.