A multimodal attention image description generation method and system

Abstract

The application discloses a kind of multimodal attention image description generation method and system, belong to image data processing technical field, the implementation of this method includes constructing bidirectional attention residual connection network and constructing attention mechanism based on dynamic attenuation factor, constructing bidirectional attention residual connection network includes: feature homogenization projection, bidirectional attention calculation, feature fusion and residual protection;Attention mechanism based on dynamic attenuation factor includes: constructing basic attenuation module based on network depth generation reference attenuation;Constructing feature sensitive module dynamically adjusted according to feature quality;Task perception module is constructed to adapt to specific task demand;Constructing gate fusion network, intelligently fusing three-way attenuation factor to generate final coefficient.This application solves the problem that image modal dominant attention weight further suppresses text modal effective information, and the problem that attention decays layer by layer due to the increase of model network depth, improves the efficiency and accuracy of image description generation.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, specifically to a multimodal attention image description generation method and system. Background Technology

[0002] Image description technology effectively ensures the semantic transformation and value extraction of visual information by transforming it into structured and semantic text descriptions. Accurate analysis of image content and generation of structured text descriptions are crucial for guaranteeing the integrity and accuracy of visual cognition, providing a reliable data foundation for subsequent applications such as visual search, content analysis, and intelligent decision-making. Therefore, efficient and accurate image description technology has significant practical and scientific value in promoting its application and development in education, security, and industrial inspection. Current research on image description generation is mainly based on the following technical approaches:

[0003] 1. Template-based methods generate sentences using a fixed template. Typically, this involves first identifying the objects in an image, their attributes, and whether they are active or passive. Then, based on this visual information, a fixed grammatical format is used as the framework for the sentence to compose the final image description.

[0004] 2. Based on the retrieval method, the system searches its own set to find the sentence that best matches the image content, based on the image content described in sentences provided by the user. The set contains some fixed image descriptions.

[0005] 3. Based on deep learning, this method utilizes a CNN network to extract image features, including objects and their attributes, and introduces additional components to extract relationships between parts. It then uses an LSTM model to decode image features and employs an attention mechanism to enhance the quality of generated image descriptions.

[0006] The above technologies mainly have the following problems:

[0007] 1. Because of the template-based approach, the syntax structure of its report is a fixed template. It simply fills in information such as specific objects, attributes and relationships into a fixed structure. Therefore, the generated image descriptions all use the same syntax structure. The syntax structure is rigid, and only keywords are replaced. The diversity of sentences is greatly limited and cannot adapt to complex scene descriptions.

[0008] 2. Since the search-based method requires a sufficient number of image reference descriptions to achieve high accuracy, and the sentences generated multiple times all come from image descriptions in the existing database, it relies on a limited sample library, resulting in a high repetition rate in the generated reports and low accuracy in rare scenarios.

[0009] 3. Deep learning-based methods that only extract features from the image modality without considering the text modality can negatively impact the accuracy of image description generation. Furthermore, existing multimodal attention mechanisms that fuse image and text modalities to generate sentences often prioritize image features over textual value. Additionally, the decay of attention across network depths can affect deep feature extraction, thus impacting report generation quality. Summary of the Invention

[0010] The technical objective of this invention is to address the above-mentioned shortcomings by providing a multimodal attention image description generation method and system. This method and system can solve the problem that image modality dominates attention weights and thus suppresses effective information from text modality, and also solve the problem that attention decays layer by layer as the model network depth increases, thereby improving the efficiency and accuracy of image description generation.

[0011] The technical solution adopted by this invention to solve its technical problem is:

[0012] A multimodal attention-based image description generation method is proposed, the implementation of which includes constructing a bidirectional attention residual connection network and constructing an attention mechanism based on a dynamic decay factor.

[0013] The construction of the bidirectional attention residual connection network includes:

[0014] Feature homogenization projection: Projecting the image feature matrix and the text feature matrix to a shared dimension space through a linear transformation;

[0015] Bidirectional attention calculation: Image-text attention uses image features as query vectors and text features as key-value pairs to calculate semantic association context; text-image attention uses text features as query vectors and image features as key-value pairs to calculate visual association context.

[0016] Feature fusion and residual preservation: Context fusion concatenates the original features with the attention context and then reduces the dimensionality; residual connection adds the fused features to the projected original features to preserve the information basis;

[0017] The construction of the attention mechanism based on the dynamic decay factor includes:

[0018] Construct a basic attenuation module based on network depth to generate baseline attenuation;

[0019] Construct a feature-sensitive module that dynamically adjusts based on feature quality;

[0020] Build a task-aware module that adapts to specific task requirements;

[0021] A gated fusion network is constructed, and the three attenuation factors are intelligently fused to generate the final coefficient.

[0022] Furthermore, the feature homogenization projection projects the image modality and text modality features into a shared dimension space by linearly transforming the image feature matrix and text feature matrix:

[0023] V′=VW v L′=LW l ;

[0024] in, Let d represent the trainable weight matrix. h To unify the dimensions of the hidden layers and align the feature space, we can ensure that modal features are interactive.

[0025] Furthermore, the image-text attention process calculates the semantic association context:

[0026]

[0027] Where Softmax represents the activation function;

[0028] The text-image attention method calculates the visual context of association.

[0029]

[0030] Furthermore, the context fusion involves concatenating the original features with the attention context and then reducing the dimensionality to obtain the fused image features F. v and fused text features F l :

[0031]

[0032] in, For linear projection layers, [||] represents the splicing operation.

[0033] Furthermore, the residual connection adds the fused features to the original projected features to obtain the updated image features O. v and updated text features O l :

[0034] O v =LayerNorm(F v +V′), O l =LayerNorm(F l +L′);

[0035] Here, LayerNorm represents the layer normalization function, and V′ and L′ constitute the information protection pathway to suppress feature degradation caused by attention noise.

[0036] The bidirectional attention residual connection network performs bidirectional attention calculation for images and text, which can effectively prevent the high image attention weight from suppressing text attention. It deeply fuses image features and text features, and performs residual connection between the fused features and the original features to effectively preserve the original features.

[0037] Furthermore, the basic attenuation module, which generates a baseline attenuation based on network depth, simulates the natural attenuation of attention as network depth increases by establishing a basic mapping relationship between network depth and attenuation intensity. The basic attenuation factor is calculated as follows:

[0038]

[0039] Where σ represents the Sigmoid activation function, β represents the learnable decay slope parameter optimized through backpropagation, α represents the minimum decay threshold to prevent excessive decay, l represents the current layer index, and L max Indicates the total number of network layers;

[0040] The constructed feature-sensitive module dynamically adjusts based on the feature quality, real-time evaluating the feature quality and dynamically adjusting the attenuation intensity. The feature-sensitive attenuation factor is calculated as follows:

[0041]

[0042] Where ReLU represents the activation function. denoted by ; W represents the feature representation of the current layer; W represents the learnable weight matrix, which maps the feature quality scalar values ​​to a new space to better fit the distribution of the decay factor; M represents the total number of feature vectors; b represents the learnable bias term, which ensures that the entire expression is not necessarily zero even when the feature quality evaluation value is small or zero, providing the model with additional degrees of adjustment freedom.

[0043] The task-aware module, adapted to specific task requirements, customizes attenuation strategies based on task type to accommodate the varying needs of different clinical tasks. Tasks such as diagnostic reports, lesion localization, and prognostic analysis are embedded into vector mappings and mapped to the attenuation strategy space. The task-aware attenuation factor is calculated as follows:

[0044]

[0045] Furthermore, the construction of the gated fusion network and the intelligent weighted three-way attenuation factor to generate the final coefficients are specifically implemented using the following formula:

[0046]

[0047] λ final =∑ k∈{base,feat,task} g k ·λ k ;

[0048] Among them, the gating weight g k Automatically learn the importance of each factor; λ k Used to represent the three-way attenuation factor: λ base , λ feat and λ task .

[0049] This invention also claims a multimodal attention-based image description generation system, comprising a bidirectional attention residual connection network construction module and an attention mechanism construction module based on a dynamic decay factor.

[0050] The construction of the bidirectional attention residual connection network includes feature homogenization projection, bidirectional attention computation, feature fusion and residual protection;

[0051] The construction of the attention mechanism based on dynamic attenuation factors includes: constructing a basic attenuation submodule that generates a baseline attenuation based on network depth; constructing a feature-sensitive submodule that dynamically adjusts according to feature quality; constructing a task-aware submodule that adapts to specific task requirements; and constructing a gated fusion network that intelligently fuses three attenuation factors to generate the final coefficient.

[0052] The system achieves multimodal attention image description generation using the methods described above.

[0053] The present invention also claims a multimodal attention image description generation apparatus, comprising: at least one memory and at least one processor;

[0054] The at least one memory is used to store a machine-readable program;

[0055] The at least one processor is used to call the machine-readable program to implement the above method.

[0056] The present invention also claims a computer-readable medium storing computer instructions that, when executed by a processor, enable the implementation of the above-described method.

[0057] Compared with the prior art, the multimodal attention image description generation method and system of the present invention have the following advantages:

[0058] The multimodal attention image description generation method provided by this invention effectively solves the problem of dominance and suppression between image modality and text modality in multimodal image description scenarios by constructing a bidirectional attention residual connection network and an attention mechanism based on dynamic decay factors. It can also solve the problem of attention decaying layer by layer as the model network depth increases. Attached Figure Description

[0059] Figure 1 This is a diagram illustrating a bidirectional attention computation structure provided in one embodiment of the present invention;

[0060] Figure 2 This is a schematic diagram of the feature fusion and residual connection structure provided in one embodiment of the present invention;

[0061] Figure 3 This is a schematic diagram of an attention mechanism structure based on a dynamic decay factor provided in an embodiment of the present invention. Detailed Implementation

[0062] The present invention will be further described below with reference to specific embodiments.

[0063] This invention provides a multimodal attention-based image caption generation method. It constructs a multimodal attention residual network, performs bidirectional attention calculations (image-text attention and text-image attention), then concatenates the original features with the attention context to obtain fused features, adds these features to the original features before projection, and performs residual connections. Next, it constructs an attention mechanism based on a dynamic decay factor, fusing three decay factors—a basic decay module, a feature-sensitive module, and a task-aware module—to generate the final coefficients. Finally, the image caption generation is guided by the multimodal attention residual network and the dynamic decay factor attention mechanism.

[0064] Its implementation includes the following steps:

[0065] (1) Construct a bidirectional attention residual connection network.

[0066] (2) Construct an attention mechanism based on dynamic decay factor.

[0067] The method for constructing a bidirectional attention residual connection network in step (1) includes: feature homogenization projection; bidirectional attention computation; feature fusion and residual protection.

[0068] The method of feature homogenization projection is to project heterogeneous modal features, namely image feature matrix and text feature matrix, onto a shared dimension space through a linear transformation.

[0069] The bidirectional attention calculation method is as follows: image-text attention uses image features as query vectors and text features as key-value pairs to calculate semantic association context; text-image attention uses text features as query vectors and image features as key-value pairs to calculate visual association context.

[0070] The method for feature fusion and residual preservation is as follows: context fusion concatenates the original features with the attention context and then reduces the dimensionality; residual connection adds the fused features to the projected original features to preserve the information basis.

[0071] The method for constructing the multimodal attention mechanism of dynamic attenuation factor in step (2) includes: constructing a basic attenuation module based on network depth to generate baseline attenuation; constructing a feature-sensitive module that is dynamically adjusted according to feature quality; constructing a task-aware module adapted to specific task requirements; and constructing a gated fusion network to intelligently fuse the three attenuation factors to generate the final coefficient.

[0072] Combination Figures 1 to 3 As shown, the specific implementation process of this method is as follows:

[0073] 1. Construct a bidirectional attention residual connection network.

[0074] First, to address the alignment issue of heterogeneous feature spaces between image and text modalities, the image and text modal features are projected to the same dimension. This is achieved by linearly transforming the image feature matrix and the text feature matrix and projecting them to a shared dimension space.

[0075] V′=VW v L′=LW l ;

[0076] in, Let d represent the trainable weight matrix. h To unify the dimensions of the hidden layers and align the feature space, we can ensure that modal features are interactive.

[0077] Then, bidirectional attention is calculated. The image focuses on the semantic context of the text, enhancing the semantic accuracy of the scene description; the text focuses on the visual context of the image, improving the accuracy of object localization. The bidirectional attention design ensures that both modalities can actively influence each other, avoiding unidirectional dominance. Its structure is as follows: Figure 1 As shown. For image-text attention, using image features as the query vector and text features as key-value pairs, the semantic association context is calculated:

[0078]

[0079] Where Softmax represents the activation function;

[0080] For text-image attention, using text features as query vectors and image features as key-value pairs, the visual context is calculated:

[0081]

[0082] Finally, to prevent attention noise from overwhelming the original features, feature fusion and residual connections are performed, with the structure as follows: Figure 2 As shown. The context fusion involves concatenating the original features with the attention context and then reducing the dimensionality to obtain the fused image features F. v and fused text features F l :

[0083]

[0084] in, For linear projection layers, [||] represents the splicing operation.

[0085] The residual connection adds the fused features to the original projected features, preserving the information basis, to obtain the updated image features O. v and updated text features O l :

[0086] O v =LayerNorm(F v +V′), O l =LayerNorm(F l +L′);

[0087] Here, LayerNorm represents the layer normalization function, and V′ and L′ constitute the information protection pathway to suppress feature degradation caused by attention noise.

[0088] The bidirectional attention residual connection network performs bidirectional attention calculation for images and text, which can effectively prevent the high image attention weight from suppressing text attention. It deeply fuses image features and text features, and performs residual connection between the fused features and the original features to effectively preserve the original features.

[0089] 2. Construct an attention mechanism based on a dynamic decay factor.

[0090] First, a basic attenuation module based on network depth is constructed. By establishing a basic mapping relationship between network depth and attenuation strength, the natural attenuation of attention as network depth increases is simulated. The basic attenuation factor is calculated as follows:

[0091]

[0092] Where σ represents the Sigmoid activation function, β represents the learnable decay slope parameter optimized through backpropagation, α represents the minimum decay threshold to prevent excessive decay, l represents the current layer index, and L max This indicates the total number of network layers.

[0093] Then, a feature-sensitive module that dynamically adjusts based on feature quality is constructed to evaluate feature quality in real time and dynamically adjust the attenuation intensity. The feature-sensitive attenuation factor is calculated as follows:

[0094]

[0095] Where ReLU represents the activation function. denoted by ; W represents the feature representation of the current layer; W represents the learnable weight matrix, which maps the feature quality scalar values ​​to a new space to better fit the distribution of the decay factor; M represents the total number of feature vectors; b represents the learnable bias term, which ensures that the entire expression is not necessarily zero even when the feature quality evaluation value is small or zero, providing the model with additional degrees of adjustment freedom.

[0096] Then, a task-aware module adapted to specific task requirements is constructed. A decay strategy is customized based on task type to adapt to the different needs of various clinical tasks. Tasks such as diagnostic reports, lesion localization, and prognostic analysis are embedded into vector mappings to the decay strategy space. The task-aware decay factor is calculated as follows:

[0097]

[0098] Finally, a gated fusion network is constructed, and the final coefficients are generated by intelligently weighting the three-way attenuation factors.

[0099]

[0100] λ final =∑k ∈{ base,feat,task}g k ·λ k ;

[0101] Among them, the gating weight g k Automatically learn the importance of each factor; λ k Used to represent the three-way attenuation factor: λ base , λ feat and λ task .

[0102] The attention mechanism based on dynamic decay factors comprehensively considers three factors: network depth, feature quality, and task-specific awareness. A decay factor module is constructed based on each factor, and intelligent fusion is achieved through a gated fusion network. Its structure is as follows: Figure 3 As shown.

[0103] This invention also provides a multimodal attention image description generation system, including a bidirectional attention residual connection network construction module and an attention mechanism construction module based on a dynamic decay factor.

[0104] The construction of the bidirectional attention residual connection network includes feature homogenization projection, bidirectional attention computation, feature fusion, and residual protection.

[0105] The method of feature homogenization projection is to project heterogeneous modal features, namely image feature matrix and text feature matrix, onto a shared dimension space through a linear transformation.

[0106] The bidirectional attention calculation method is as follows: image-text attention uses image features as query vectors and text features as key-value pairs to calculate semantic association context; text-image attention uses text features as query vectors and image features as key-value pairs to calculate visual association context.

[0107] The method for feature fusion and residual preservation is as follows: context fusion concatenates the original features with the attention context and then reduces the dimensionality; residual connection adds the fused features to the projected original features to preserve the information basis.

[0108] The construction of the attention mechanism based on dynamic attenuation factors includes: constructing a basic attenuation submodule that generates a baseline attenuation based on network depth; constructing a feature-sensitive submodule that dynamically adjusts according to feature quality; constructing a task-aware submodule that adapts to specific task requirements; and constructing a gated fusion network that intelligently fuses three attenuation factors to generate the final coefficient.

[0109] Specifically, the system achieves multimodal attention image description generation through the multimodal attention image description generation method described in the above embodiments.

[0110] This invention also provides a multimodal attention image description generation apparatus, comprising: at least one memory and at least one processor;

[0111] The at least one memory is used to store a machine-readable program;

[0112] The at least one processor is used to call the machine-readable program to implement the multimodal attention image description generation method described in the above embodiments.

[0113] This invention also provides a computer-readable medium storing computer instructions that, when executed by a processor, implement the multimodal attention image description generation method described in the above embodiments. Specifically, a system or apparatus equipped with a storage medium storing software program code that implements the functions of any of the embodiments described above, and enabling the computer (or CPU or MPU) of the system or apparatus to read and execute the program code stored in the storage medium.

[0114] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0115] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0116] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0117] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0118] The present invention has been shown and described in detail above with reference to the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above embodiments, those skilled in the art will know that more embodiments of the present invention can be obtained by combining the code review methods in the different embodiments. These embodiments are also within the protection scope of the present invention.

Claims

1. A method for generating multimodal attention-based image descriptions, characterized in that, The implementation of this method includes constructing a bidirectional attention residual connection network and constructing an attention mechanism based on a dynamic decay factor. The construction of the bidirectional attention residual connection network includes: Feature homogenization projection: Projecting the image feature matrix and the text feature matrix to a shared dimension space through a linear transformation; Bidirectional attention calculation: Image-text attention uses image features as query vectors and text features as key-value pairs to calculate semantic association context; text-image attention uses text features as query vectors and image features as key-value pairs to calculate visual association context. Feature fusion and residual preservation: Context fusion concatenates the original features with the attention context and then reduces the dimensionality; residual connection adds the fused features to the projected original features to preserve the information basis; The construction of the attention mechanism based on the dynamic decay factor includes: Construct a basic attenuation module based on network depth to generate baseline attenuation; Construct a feature-sensitive module that dynamically adjusts based on feature quality; Build a task-aware module that adapts to specific task requirements; Construct a gated fusion network and intelligently fuse three attenuation factors to generate the final coefficient; The basic attenuation module, which generates a baseline attenuation based on network depth, simulates the natural attenuation of attention as network depth increases by establishing a basic mapping relationship between network depth and attenuation intensity. The basic attenuation factor is calculated as follows: ； in, This represents the Sigmoid activation function. This indicates that the decay slope parameter can be learned and optimized through backpropagation. This indicates the minimum attenuation threshold to prevent excessive attenuation. Indicates the index of the current layer. Indicates the total number of network layers; The constructed feature-sensitive module dynamically adjusts based on the feature quality, real-time evaluating the feature quality and dynamically adjusting the attenuation intensity. The feature-sensitive attenuation factor is calculated as follows: ； Where ReLU represents the activation function. This represents the feature representation of the current layer. W This represents the learnable weight matrix. M This represents the total number of eigenvectors. i Indicates the index of the feature vector. b Represents a learnable bias term; The task-aware module, adapted to specific task requirements, customizes attenuation strategies based on task type to accommodate the differences in needs across various clinical tasks. The task embedding vector is mapped to the attenuation strategy space, and the task-aware attenuation factor is calculated as follows: ； in, This represents a multilayer perceptron. Represents the task embedding vector. This represents a vector concatenation operation; The constructed gated fusion network uses intelligent weighted three-way attenuation factors to generate the final coefficients, and the specific implementation formula is as follows: ； Among them, gating weights Automatically learn the importance of each factor; for k Path attenuation factor, where k Pick base, feat, task One of the three.

2. The multimodal attention image description generation method according to claim 1, characterized in that, The feature homogenization projection projects the image modality and text modality features into a shared dimension space through a linear transformation: ； in, This represents a trainable weight matrix. To unify the dimensions of the hidden layers and align the feature space, we can ensure that modal features are interactive.

3. The multimodal attention image description generation method according to claim 2, characterized in that, The image-text attention method calculates semantic association context: ； Where Softmax represents the activation function; The text-image attention method calculates the visual context of association. 。 4. The multimodal attention image description generation method according to claim 3, characterized in that, The context fusion process involves concatenating the original features with the attention context and then reducing the dimensionality to obtain fused image features. and fusion text features : ； in, For linear projection layers, [||] represents the splicing operation.

5. The multimodal attention image description generation method according to claim 4, characterized in that, The residual connection adds the fused features to the original projected features to obtain the updated image features. and updated text features : ； Where LayerNorm represents the layer normalization function. and It constitutes an information protection pathway to suppress feature degradation caused by attention noise.

6. A multimodal attention-based image description generation system, characterized in that, It includes a bidirectional attention residual connection network construction module and an attention mechanism construction module based on dynamic decay factor. The bidirectional attention residual connection network is constructed. This includes feature homogenization projection, bidirectional attention computation, feature fusion, and residual preservation; The construction of the attention mechanism based on dynamic attenuation factors includes: constructing a basic attenuation submodule that generates a baseline attenuation based on network depth; constructing a feature-sensitive submodule that dynamically adjusts according to feature quality; constructing a task-aware submodule that adapts to specific task requirements; and constructing a gated fusion network that intelligently fuses three attenuation factors to generate the final coefficient. The system achieves multimodal attention image description generation by means of any one of claims 1 to 5.

7. A multimodal attention-based image description generation device, characterized in that, include: At least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to invoke the machine-readable program to implement the method of any one of claims 1 to 5.

8. A computer-readable medium, characterized in that, The computer-readable medium stores computer instructions that, when executed by a processor, enable the implementation of the method described in any one of claims 1 to 5.

Images