Text generation method based on dual attention multi-modal fusion

Abstract

The invention relates to the technical field of natural language processing, in particular to a text generation method based on dual attention multi-modal fusion, which comprises the following steps: acquiring image features by using a visual feature extractor, and performing semantic coding on an optional input text through a text embedding module; inputting the visual features into a double attention enhancement module, and performing parallel processing by position attention and channel attention branches; the visual features, the output of the attention module and the text embedding result are fused and sent to a Transform encoder for depth representation; historical information is maintained in combination with a guide memory generator, and decoder parameters are dynamically adjusted through a context guide normalization mechanism; and finally, generating a text in a decoder through autoregression and cross attention. According to the method, space and channel dimension features are considered in multi-modal information fusion, the generation accuracy and continuity are improved, and the problem of poor text generation quality in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a text generation method based on dual attention multimodal fusion. Background Technology

[0002] With the rapid development of artificial intelligence technology and the explosive growth of multimedia data, traditional single-modal text generation methods can no longer meet the needs of complex application scenarios. Existing text generation technologies mainly rely on single text or image information for content generation, lacking effective fusion and utilization of multimodal information, resulting in insufficient accuracy and richness of the generated text content.

[0003] Multimodal fusion text generation methods aim to fully utilize the complementary information between visual and textual features, achieving effective integration of cross-modal information through deep learning techniques. Traditional multimodal fusion methods typically employ simple feature concatenation or weighted fusion strategies, failing to fully explore the deep semantic relationships between different modalities. Furthermore, existing methods often suffer from limitations in handling complex scenarios, such as simplistic attention mechanisms and limited feature representation capabilities, making it difficult to accurately capture key information and generate high-quality descriptive text. This limitation is particularly pronounced in specialized fields like medical image diagnostic report generation, where the accuracy and professionalism of text generation are extremely high, highlighting the shortcomings of traditional methods and resulting in poor text generation quality in existing technologies. Summary of the Invention

[0004] The purpose of this invention is to provide a text generation method based on dual attention multimodal fusion, which aims to solve the problem of poor text generation quality in the prior art.

[0005] To achieve the above objectives, this invention provides a text generation method based on dual attention multimodal fusion, comprising: The input image is processed by a visual feature extractor to extract features, generate multi-scale visual feature representations, and establish a hierarchical feature structure for the image content. The optional input text is semantically encoded through the text embedding module, the text semantic features are extracted using a pre-trained vision-language model, and multimodal fusion is achieved through dimensional projection and visual features. The visual feature representation is input into the dual attention enhancement module, and spatial relationship and channel dependency information are extracted through parallel processing of position attention branch and channel attention branch; The visual feature representation, the output of the dual attention module, and the output of the text embedding module are fused together. A multimodal fusion feature is generated through feature summation. This fusion feature is then input into the Transformer encoder for deep encoding to generate a hidden state representation containing rich semantic information. Based on the historical information of the generation process maintained by the guided memory generator, the normalization parameters of the decoder are dynamically adjusted through the context-guided normalization mechanism to achieve adaptive text generation; Autoregressive sequence generation is performed through a decoder, and high-quality structured text content is generated by interacting with the encoder output through a cross-attention mechanism.

[0006] The specific steps for extracting features from the input image using a visual feature extractor include: The input image is preprocessed, including size normalization, numerical normalization and data augmentation. ResNet-101 is used as the backbone network, and global features and local detail features of the image are extracted through convolutional neural networks to generate feature maps at multiple scale levels.

[0007] The specific steps involved in semantically encoding the optional input text through a text embedding module, extracting semantic features from the text using a pre-trained vision-language model, and achieving multimodal fusion through dimensional projection and visual features include: The input text is processed by the pre-trained CLIP model's text encoder, and the input text is tokenized using a CLIP-specific word segmenter. Semantic feature representations are extracted through the text encoder and L2 normalization is applied. Multimodal feature fusion is achieved by extending the text feature vector to the spatial dimension of visual features and adding it element-wise with the visual features.

[0008] In the step of inputting the visual feature representation into the dual attention enhancement module, and extracting spatial relationship and channel dependency information through parallel processing of positional attention branch and channel attention branch, The dual attention enhancement module processes spatial relationships and channel dependencies in parallel through positional attention and channel attention branches. The positional attention branch calculates the product of the input feature matrix and its transpose, normalizes it using a scaling factor, and generates a spatial attention map using a softmax function. The channel attention branch generates a channel attention map by calculating the product of the transpose of the feature matrix and the original matrix. The positional attention branch and the channel attention branch are integrated using a weighted fusion strategy.

[0009] The location attention branch calculates the product of the input feature matrix and its transpose, standardizes it using a scaling factor, and then applies the softmax function to generate the spatial attention map, including: The positional attention branch is applied to the input feature matrix. To compute the spatial attention map, we first normalize it by multiplying matrix A and its transpose using a scaling factor. The calculation formula is as follows: ; in Using the channel dimension, this spatial attention map can capture the correlation between different spatial locations in an image, enhancing the feature representation of important regions. The output is as follows: ; in These are learnable weight parameters.

[0010] The channel attention branch generates the channel attention map by multiplying the transpose of the feature matrix with the original matrix, including: The channel attention branch generates the channel attention map by multiplying the transpose of the feature matrix by the original matrix. The calculation formula is as follows: ; This attention map can model the dependencies between different feature channels, highlighting feature channels that are important to the current task. The output is as follows: ; in These are learnable weight parameters.

[0011] The location attention branch and channel attention branch are integrated using a weighted fusion strategy, including: The positional attention branch and the channel attention branch are integrated using a weighted fusion strategy, and the final output is: ; in To integrate weight parameters, , Initialized to 0.1, Setting it to 0.5 enables synergistic enhancement of features in both spatial and channel dimensions.

[0012] The step of fusing the visual feature representation, the output of the dual attention module, and the output of the text embedding module, generating a multimodal fusion feature through feature summation, and then inputting this fusion feature into the Transformer encoder for deep encoding to generate a hidden state representation containing rich semantic information includes: The enhanced feature sequence is mapped to a unified hidden dimension space through a linear projection layer; the visual feature sequence is deeply encoded through a multi-layer Transformer encoder, each layer of which includes a multi-head self-attention mechanism, positional encoding, residual connections, and layer normalization techniques.

[0013] Among them, adaptive text generation is achieved by maintaining historical information of the generation process based on the guided memory generator and dynamically adjusting the normalization parameters of the decoder through a context-guided normalization mechanism, including: The guided memory generator employs a multi-head attention mechanism to maintain historical information during the generation process. In the guided memory update mechanism, the guided memory from the previous time step is used. The guided memory will be concatenated with the text output generated in the previous step, serving as the query vector Q. As key K and value V, the attention output is obtained through multi-head attention computation: ; Subsequently, residual connections and multilayer perceptron processing are performed, and the calculation formula is as follows: ; Gating mechanisms control the flow of information, including forget gates: ; Input gate: ; in This represents the sigmoid activation function. The weights and bias parameters of the forget gate and input gate are respectively given. The formula for guiding memory updates is as follows: ; in This represents the Hadamard product, which is the multiplication of corresponding elements in a matrix. To ensure training stability, an exponential moving average strategy is used to smoothly update the bootstrap memory. The calculation formula is as follows: ; Among the momentum parameters Set to 0.99; The context-guided normalization mechanism dynamically generates incremental values ​​of normalization parameters based on the current guidance memory, and maps the guidance memory to a multilayer perceptron. and The adjustment amount of the parameter is calculated using the following formula: ; ; During dynamic normalization, the basic learnable parameters are added to the incremental values ​​generated by the multilayer perceptron to obtain the dynamic parameters: ; ; Then apply the context-guided normalization formula. ,in As input features, and These are the mean and standard deviation of the input features, respectively.

[0014] In the step of generating high-quality structured text content by generating autoregressive sequences through a decoder and interacting with the encoder output using a cross-attention mechanism, the autoregressive sequence generation by the decoder includes: The decoder employs a masked multi-head self-attention mechanism to achieve autoregressive generation, ensuring that it can only access information prior to the current position during the generation process, thus maintaining the causality of the generation. A cross-attention layer interacts the decoder's hidden state with the encoder output, enabling the decoder to effectively utilize visual encoding information to guide the text generation process. A context-guided normalization mechanism is integrated into each decoding layer, dynamically adjusting the feature distribution based on the current generation context to ensure semantic consistency between the generated content and the visual input. The decoder output passes through a linear projection layer and a softmax function to obtain a probability distribution over the vocabulary. A top-k search strategy is used to select the next generated word, gradually constructing a complete text sequence. The entire generation process is jointly optimized using multiple loss functions, including cross-entropy loss, semantic similarity loss, and domain-specific terminology consistency loss.

[0015] This invention discloses a text generation method based on dual-attention multimodal fusion. In the dual-attention enhancement module, parallel processing of positional and channel attention branches simultaneously captures the spatial relationships and channel feature dependencies of images, fully extracting key information features from visual content. In the context-guided normalization mechanism, the normalization parameters are dynamically adjusted using historical information maintained by the guided memory generator, achieving adaptive responses to different generation contexts. In the text embedding module, the powerful semantic understanding capabilities of a pre-trained visual-language model are utilized to achieve effective fusion of text and visual modalities, providing semantic guidance for the generation process. This invention fully extracts spatial and channel dimension information from visual features and effectively fuses the semantic content of multimodal inputs, significantly improving the accuracy and coherence of text generation. It solves the problems of incomplete feature representation, poor context adaptability, and unstable generation quality in traditional methods, thus addressing the issue of poor text generation quality in existing technologies. This effectively improves the generation effect and generalization ability of multimodal text generation systems in complex application scenarios, providing reliable technical support for fields such as medical image diagnosis, image content understanding, and multimedia analysis. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0017] Figure 1This is a flowchart of the text generation method based on dual attention multimodal fusion of the present invention.

[0018] Figure 2 This is a network structure diagram of the text generation method based on dual attention multimodal fusion of the present invention.

[0019] Figure 3 This is a schematic diagram of the dual attention enhancement module of the text generation method based on dual attention multimodal fusion of the present invention.

[0020] Figure 4 This is a schematic diagram of the guided memory generation module of the text generation method based on dual attention multimodal fusion of the present invention.

[0021] Figure 5 This is a schematic diagram of the decoder structure of the text generation method based on dual attention multimodal fusion of the present invention. Detailed Implementation

[0022] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0023] Please see Figures 1-5 ,in, Figure 1 This is a flowchart of the text generation method based on dual attention multimodal fusion of the present invention; Figure 2 This is a network structure diagram of the text generation method based on dual attention multimodal fusion of the present invention; Figure 3 This is a schematic diagram of the dual attention enhancement module of the text generation method based on dual attention multimodal fusion of the present invention; Figure 4 This is a schematic diagram of the guided memory generation module of the text generation method based on dual attention multimodal fusion of the present invention; Figure 5 This is a schematic diagram of the decoder structure of the text generation method based on dual attention multimodal fusion of the present invention.

[0024] This invention provides a text generation method based on dual attention multimodal fusion, comprising: S1 extracts features from the input image using a visual feature extractor, generates multi-scale visual feature representations, and establishes a hierarchical feature structure for the image content. The specific steps for extracting features from an input image using a visual feature extractor include: The input images undergo preprocessing operations, including size normalization, numerical normalization, and data augmentation. ResNet-101 is used as the backbone network, and a convolutional neural network is employed to extract global and local detail features, generating feature maps at multiple scale levels. A complete representation system from low-level visual features to high-level semantic features is established, providing a rich visual information foundation for subsequent attention calculations and text generation.

[0025] In this embodiment of the invention, the input medical image is first preprocessed, including standardizing the image size to 224×224 pixels, normalizing pixel values ​​to the range of 0 to 1, and applying data augmentation techniques such as random horizontal flipping, random rotation by ±5 degrees, and brightness and contrast adjustment. An improved ResNet-101 is used as the backbone network for the visual feature extractor, extracting global and local detail features of the image through a convolutional neural network. The network structure includes an initial convolutional layer with 7×7 kernels, a stride of 2, and 64 output channels, followed by four residual block groups containing 3, 4, 23, and 3 residual units respectively. Finally, an average pooling layer downsamples the feature map to a fixed size, and a fully connected layer outputs a 2048-dimensional feature vector. Through a multi-scale feature extraction strategy, features are extracted from different levels of the network, generating feature maps at multiple scales, establishing a complete representation system from low-level visual features such as edges and textures to high-level semantic features such as organ structure and pathological features.

[0026] S2 semantically encodes the optional input text through a text embedding module, extracts semantic features from the text using a pre-trained vision-language model, and achieves multimodal fusion through dimensionality projection and visual features; specific steps include: The input text is processed by the pre-trained CLIP model's text encoder, and the input text is tokenized using a CLIP-specific word segmenter. Semantic feature representations are extracted through the text encoder and L2 normalization is applied. Multimodal feature fusion is achieved by extending the text feature vector to the spatial dimension of visual features and adding it element-wise with the visual features.

[0027] In this embodiment of the invention, when the system receives optional input text prompts, it employs the ViT-B / 32 architecture of the pre-trained CLIP model as the text encoder. This encoder possesses powerful text semantic understanding capabilities and visual-language alignment capabilities. The text processing flow first uses a CLIP-specific tokenizer to tokenize the input text, truncating the text sequence to a maximum length of 60 tokens, with zero-padding for any insufficient length. Then, the CLIP text encoder extracts semantic feature representations (512-dimensional semantic feature representations). These are then expanded to 2048 dimensions through a linear projection layer to match the visual feature dimensions. Finally, L2 normalization is applied to ensure the feature vectors lie on a unit sphere, improving the stability of numerical computation. Multimodal feature fusion is achieved through a broadcast mechanism, expanding the text feature vectors to the spatial dimension of visual features and adding them element-wise with the visual features. This allows text semantic information to effectively guide the learning of visual feature representations, enhancing the model's comprehensive understanding of the input content.

[0028] S3 inputs the visual feature representation into the dual attention enhancement module, and extracts spatial relationship and channel dependency information through parallel processing of position attention branch and channel attention branch; In this step, the dual attention enhancement module processes spatial relationships and channel dependencies in parallel through positional attention and channel attention branches. Specifically, the positional attention branch calculates the product of the input feature matrix and its transpose, normalizes it using a scaling factor, and generates a spatial attention map using the softmax function. The channel attention branch generates a channel attention map by calculating the product of the transpose of the feature matrix and the original matrix. The positional attention branch and the channel attention branch are integrated using a weighted fusion strategy.

[0029] like Figure 3 As shown, specifically, the location attention branch calculates the product of the input feature matrix and its transpose, performs normalization using a scaling factor, and applies the softmax function to generate the spatial attention map, including: The positional attention branch is applied to the input feature matrix. To compute the spatial attention map (SAG), we first normalize it by multiplying matrix A and its transpose using a scaling factor. The calculation formula is as follows: ; in Using the channel dimension, this spatial attention map can capture the correlation between different spatial locations in an image, enhancing the feature representation of important regions. The output is as follows: ; in These are learnable weight parameters.

[0030] Specifically, the channel attention branch generates the channel attention map by calculating the product of the transpose of the feature matrix and the original matrix, including: The channel attention branch generates the channel attention map (CAG) by multiplying the transpose of the feature matrix by the original matrix. The calculation formula is as follows: ; This attention map can model the dependencies between different feature channels, highlighting feature channels that are important to the current task. The output is as follows: ; in These are learnable weight parameters.

[0031] Specifically, the location attention branch and channel attention branch are integrated through a weighted fusion strategy, including: The positional attention branch and the channel attention branch are integrated using a weighted fusion strategy, and the final output is: ; in To integrate weight parameters, , Initialized to 0.1, Setting it to 0.5 enables synergistic enhancement of features in both spatial and channel dimensions.

[0032] S4 fuses the visual feature representation, the output of the dual attention module, and the output of the text embedding module, generates a multimodal fusion feature through feature summation, and then inputs the fusion feature into the Transformer encoder for deep encoding to generate a hidden state representation containing rich semantic information. In this step, the deep encoding of the Transformer encoder includes: mapping the enhanced feature sequence to a unified hidden dimension space through a linear projection layer; and performing deep encoding of the visual feature sequence through a multi-layer Transformer encoder, where each layer includes a multi-head self-attention mechanism, positional encoding, residual connections, and layer normalization techniques. Each layer of self-attention can capture the dependencies between elements at different positions in the sequence, gradually improving the abstraction level and semantic richness of the feature representation. Residual connections and layer normalization techniques are used during the encoding process to ensure the training stability and gradient propagation effect of the deep network, ultimately generating the encoder hidden state output containing global semantic information.

[0033] In this embodiment of the invention, the enhanced feature sequence is mapped to a unified 512-dimensional hidden dimension space through a linear projection layer to ensure the consistency of feature dimensions. A multi-layer Transformer encoder with three layers is used to perform deep encoding on the visual feature sequence. Each encoder layer includes a multi-head self-attention mechanism that uses eight attention heads to capture long-distance dependencies between elements at different positions in the sequence. Sine position encoding is used to provide positional information for the feature sequence. Residual connections and layer normalization ensure the training stability of the deep network. A feedforward neural network with a hidden layer dimension of 2048 performs nonlinear feature transformation. The encoding process gradually improves the abstraction level and semantic richness of the feature representation, ultimately generating the encoder hidden state output containing global semantic information.

[0034] S5 maintains historical information about the generation process based on a guided memory generator, and dynamically adjusts the normalization parameters of the decoder through a context-guided normalization mechanism to achieve adaptive text generation; specifically including: The guided memory generator employs a multi-head attention mechanism to maintain historical information during the generation process. In the guided memory update mechanism, the guided memory from the previous time step is used. The guided memory will be concatenated with the text output generated in the previous step, serving as the query vector Q. As key K and value V, the attention output is obtained through multi-head attention computation: ; Subsequently, residual connections and multilayer perceptron processing are performed, and the calculation formula is as follows: ; Gating mechanisms control the flow of information, including forget gates: ; Input gate: ; in This represents the sigmoid activation function. The weights and bias parameters of the forget gate and input gate are respectively given. The formula for guiding memory updates is as follows: ; in This represents the Hadamard product, which is the multiplication of corresponding elements in a matrix. To ensure training stability, an exponential moving average strategy is used to smoothly update the bootstrap memory. The calculation formula is as follows: ; Among the momentum parameters Set to 0.99; The context-guided normalization mechanism dynamically generates incremental values ​​of normalization parameters based on the current guidance memory, and maps the guidance memory to a multilayer perceptron. and The adjustment amount of the parameter is calculated using the following formula: ; ; During dynamic normalization, the basic learnable parameters are added to the incremental values ​​generated by the multilayer perceptron to obtain the dynamic parameters: ; ; Then apply the context-guided normalization formula. ,in As input features, and These are the mean and standard deviation of the input features, respectively.

[0035] S6 generates autoregressive sequences through a decoder and interacts with the encoder output using a cross-attention mechanism to produce high-quality structured text content.

[0036] In this step, the decoder employs a masked multi-head self-attention mechanism to achieve autoregressive generation, ensuring that it can only access information prior to the current position during the generation process, thus maintaining the causality of the generation. A cross-attention layer interacts the decoder's hidden state with the encoder output, enabling the decoder to effectively utilize visual encoding information to guide the text generation process. A context-guided normalization mechanism is integrated into each decoding layer, dynamically adjusting the feature distribution based on the current generation context to ensure semantic consistency between the generated content and the visual input. The decoder output passes through a linear projection layer and a softmax function to obtain the probability distribution over the vocabulary. A top-k search strategy is used to select the next generated word, gradually constructing a complete text sequence. The entire generation process is jointly optimized using multiple loss functions, including cross-entropy loss, semantic similarity loss, and domain-specific terminology consistency loss, ensuring the quality and accuracy of the generated text.

[0037] In embodiments of the present invention, such as Figure 2 and Figure 5As shown, the decoder employs a Transformer architecture to achieve autoregressive text generation, comprising three decoding layers. Each layer includes a masked multi-head self-attention layer, a cross-attention layer, a context-guided normalization layer, and a feedforward network. The masked multi-head self-attention layer uses a lower triangular mask matrix to ensure the causality of the generated text. During training, the system employs a teacher-forcing strategy. The decoder's masked multi-head self-attention layer uses the real label sequence as input; that is, at time step t, the decoder can access all real label information from the beginning of the sequence to the current position t. The masked multi-head self-attention layer ensures causal constraints during training through a lower triangular mask matrix, calculated as follows: ; in The matrix is ​​a lower triangular matrix to prevent future information leakage. During the inference phase, the decoder uses an autoregressive generation method, with the output predicted by the model at the previous time step serving as the input for the current time step, progressively constructing a complete text sequence. The cross-attention layer uses the decoder's hidden state as the query Q and the encoder output as the key K and value V. The cross-attention calculation formula is: ; This enables interaction between visual information and text generation. The context-guided normalization layer integrates the normalization mechanism from step S5, dynamically adjusting the feature distribution based on the generation context. The feedforward network performs feature transformation and nonlinear mapping; the calculation formula is as follows: ; The final output of the decoder first undergoes a dimensionality transformation through a fully connected layer, changing the dimension of the hidden state from... Mapping to vocabulary size The calculation formula is: ; in The output is the hidden state of the decoder. and These represent the weights and bias parameters of the fully connected layer. Then, the logits are normalized using the softmax function to generate the probability distribution over the vocabulary, calculated as follows: ; Based on this probability distribution, a top-k sampling strategy is used for lexical selection, with k set to 50. Complete text sequences are gradually constructed until an end marker is generated or the maximum sequence length is reached. The system employs a multi-task joint loss function for end-to-end training, including cross-entropy loss, semantic similarity loss, and domain-specific term label consistency loss. The joint loss calculation formula is as follows: ; Cross-entropy loss , For real tag terms, weight Set to 1.0; semantic similarity loss semantic similarity calculation based on CLIP, weights Set to 0.2; Clinical label consistency loss Classification accuracy based on domain-specific terms, weighting Set it to 0.2.

[0038] This invention presents a text generation method based on dual-attention multimodal fusion. In the dual-attention enhancement module, parallel processing of positional and channel attention branches simultaneously captures the spatial relationships and channel feature dependencies of images, fully extracting key information features from visual content. In the context-guided normalization mechanism, the normalization parameters are dynamically adjusted using historical information maintained by the guided memory generator, achieving adaptive responses to different generation contexts. In the text embedding module, the powerful semantic understanding capabilities of a pre-trained visual-language model are utilized to achieve effective fusion of text and visual modalities, providing semantic guidance for the generation process. This invention fully extracts spatial and channel dimension information from visual features and effectively fuses the semantic content of multimodal inputs, significantly improving the accuracy and coherence of text generation. It solves the problems of incomplete feature representation, poor context adaptability, and unstable generation quality in traditional methods, thus addressing the issue of poor text generation quality in existing technologies. This effectively improves the generation effect and generalization ability of multimodal text generation systems in complex application scenarios, providing reliable technical support for fields such as medical image diagnosis, image content understanding, and multimedia analysis.

[0039] The above-disclosed embodiments are merely one or more preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments and equivalent changes made in accordance with the claims of this application still fall within the scope of this application.

Claims

1. A text generation method based on dual attention multi-modal fusion, characterized in that, The application relates to a method for generating structured text based on image content, comprising the following steps: input image is extracted by a visual feature extractor to generate multi-scale visual feature representation, and a hierarchical feature structure of image content is established; optional input text is processed by a text embedding module to extract text semantic features by using a pre-trained visual-linguistic model, and multi-modal fusion is realized by dimension projection and visual features; the visual feature representation is input into a dual attention enhancement module, and spatial relationship and channel dependency information are extracted by parallel processing of a position attention branch and a channel attention branch; the visual feature representation, the output of the dual attention module and the output of the text embedding module are fused to generate multi-modal fusion features by feature summation operation, and then the fusion features are input into a Transformer encoder for deep coding to generate hidden state representation containing rich semantic information; the guided memory generator is used to maintain the history information of the generation process, and the context-guided normalization mechanism is used to dynamically adjust the normalization parameters of the decoder to realize adaptive text generation; self-attention mechanism is combined with the output of the encoder to generate high-quality structured text content.

2. The dual attention based multi-modal fusion text generation method of claim 1, wherein, The specific steps of extracting the input image by the visual feature extractor include: preprocessing operations are performed on the input image, including size standardization, value normalization and data enhancement processing; ResNet-101 is used as a backbone network, and convolutional neural networks are used to extract global features and local detail features of the image to generate multiple scale-level feature maps.

3. The dual attention based multi-modal fusion text generation method of claim 1, wherein, The specific steps of processing the optional input text by the text embedding module to extract text semantic features by using a pre-trained visual-linguistic model and realizing multi-modal fusion by dimension projection and visual features include: the input text is processed by the text encoder of the pre-trained CLIP model, the input text is tokenized by using a CLIP special tokenizer, the semantic feature representation is extracted by the text encoder, and L2 normalization processing is applied; multi-modal feature fusion is realized by expanding the text feature vector to the spatial dimension of the visual feature and performing element-wise addition with the visual feature.

4. The dual attention based multi-modal fusion text generation method of claim 1, wherein, In the step of inputting the visual feature representation into the dual attention enhancement module and extracting spatial relationship and channel dependency information by parallel processing of the position attention branch and the channel attention branch, the dual attention enhancement module processes the position attention branch and the channel attention branch in parallel to extract spatial relationship and channel dependency information respectively; wherein, the position attention branch calculates the matrix product of the input feature matrix and its transpose, performs scaling factor standardization processing, and generates a spatial attention map by using a softmax function; the channel attention branch generates a channel attention map by calculating the product of the transpose of the feature matrix and the original matrix; the position attention branch and the channel attention branch are integrated by a weighted fusion strategy.

5. The dual attention based multi-modal fusion text generation method of claim 4, wherein, The position attention branch calculates the matrix product of the input feature matrix and its transpose, performs scaling factor standardization processing, and generates a spatial attention map by using a softmax function, which includes: The position attention branch is applied to the input feature matrix The spatial attention map is calculated first by matrix A multiplied by its transpose, normalized by a scaling factor, with the formula: ； wherein For the channel dimension, the spatial attention map can capture the correlation between different spatial positions in the image, enhance the feature representation of important regions, and output the result as: ； wherein are learnable weight parameters.

6. The dual attention based multi-modal fusion text generation method of claim 5, wherein, The channel attention branch generates a channel attention map by calculating the product of the transpose of the feature matrix and the original matrix, including: The channel attention branch generates a channel attention map by calculating the product of the transpose of the feature matrix and the original matrix, and the calculation formula is: ； This attention map can model the dependency between different feature channels, highlight the feature channels important to the current task, and the output result is: ； wherein are learnable weight parameters.

7. The dual attention based multi-modal fusion text generation method of claim 6, wherein, The position attention branch and the channel attention branch are integrated through a weighted fusion strategy, including: The position attention branch and the channel attention branch are integrated through a weighted fusion strategy, and the final output is: ； wherein is a fusion weight parameter, , initialized to 0.1, set to 0.5, enabling synergistic enhancement of spatial and channel dimension features.

8. The dual attention based multi-modal fusion text generation method of claim 1, wherein, In the step of fusing the visual feature representation, the output of the dual attention module, and the output of the text embedding module, generating multi-modal fusion features through feature summation operation, and then inputting the fusion features into the Transformer encoder for deep encoding to generate hidden state representation containing rich semantic information, the Transformer encoder deep encoding includes: The enhanced feature sequence is mapped to a unified hidden dimension space through a linear projection layer; the visual feature sequence is deep encoded through a multi-layer Transformer encoder, and each layer of the encoder includes multi-head self-attention mechanism, position encoding, residual connection, and layer normalization technology.

9. The dual attention based multi-modal fusion text generation method of claim 1, wherein, The guided memory generator maintains the historical information of the generation process, dynamically adjusts the normalization parameters of the decoder through the context-guided normalization mechanism, and realizes adaptive text generation, including: The guiding memory generator adopts a multi-head attention mechanism to maintain the historical information of the generation process. In the guiding memory update mechanism, the guiding memory of the previous time step is used As the query vector Q, the guiding memory is concatenated with the text output generated in the previous step As the key K and the value V, the attention output is obtained by multi-head attention calculation: ； Then, residual connection and multi-layer perceptron processing are performed, and the calculation formula is: ； The gating mechanism controls information flow, including a forget gate: ； and an input gate: ； wherein denotes a sigmoid activation function, are the weight and bias parameters of the forget and input gates, respectively, and the update formula for the memory is ； wherein represents a Hadamard product, i.e. multiplication of corresponding elements of matrices; To ensure training stability, an exponential moving average strategy is used to update the guided memory smoothly, and the calculation formula is: ； where the momentum parameter is set to 0.99; The context-guided normalization mechanism dynamically generates an incremental value of the normalization parameter according to the current guiding memory, and maps the guiding memory into and The adjustment amount of the parameter is calculated by the following formula: ； ； In the dynamic normalization process, the basic learnable parameter is added to the incremental value generated by the multi-layer perceptron to obtain the dynamic parameter: ； ； The context-guided normalization formula is then applied where is the input feature, and are the mean and standard deviation of the input feature, respectively.

10. The dual attention based multi-modal fusion text generation method of claim 1, wherein, In the step of generating an autoregressive sequence through the decoder, combining the cross-attention mechanism and interacting with the encoder output to generate high-quality structured text content, the decoder generates an autoregressive sequence, including: The decoder uses a masked multi-head self-attention mechanism to realize autoregressive generation, ensuring that only information before the current position can be accessed during the generation process, maintaining the causality of the generation; the cross-attention layer interacts the decoder's hidden state with the encoder output, enabling the decoder to effectively utilize visual encoding information to guide the text generation process; the context-guided normalization mechanism is integrated in each decoding layer to dynamically adjust the feature distribution based on the current generation context, ensuring the semantic consistency of the generated content with the visual input; the decoder output is projected through a linear projection layer and a softmax function to obtain a probability distribution on the vocabulary, and the next generated word is selected through a top-k search strategy, gradually constructing a complete text sequence; the entire generation process is jointly optimized through multiple loss functions, including cross-entropy loss, semantic similarity loss, and consistency loss of task-specific terminology.

Images