Image text description generation method and system based on anchor point visual feature diffusion

Abstract

This invention discloses an image text description generation method and system based on anchor point visual feature diffusion. The method includes extracting global and local features from the input image; reshaping and multi-directional encoding the local features in two-dimensional space, followed by anchor point visual feature diffusion to obtain enhanced local features; cross-inserting the global features with the original local features, and then using a bidirectional Mamba encoder for bidirectional context modeling to obtain enhanced global features; fusing the enhanced local and global features to generate a unified visual representation; and progressively decoding the unified visual representation to generate an image text description. This invention aims to address the technical problems of weak local structural feature representation, loss of positional information in global features, and lack of adaptability in global-local feature fusion in existing image text description methods, thereby improving the accuracy, semantic richness, and scene matching of image text descriptions.

Description

Technical Field

[0001] This invention relates to the field of image recognition or understanding technology, and specifically to an image text description generation method and system based on anchor point visual feature diffusion. Background Technology

[0002] Image text description technology can transform the spatial layout and semantic content of images into natural language descriptions, which has significant application value in fields such as human-computer interaction, security monitoring, and medical education. However, due to the problems of natural images, such as cluttered spatial layout, large target scale variations, and weak correlation between local features and global semantics, traditional image description methods based on CLIP visual encoders, which divide the image into regular patches and only model feature interactions through basic visual Transformers, tend to split complete target regions into different patches, resulting in weakened representation of local structural features. At the same time, the global features of CLS (Category Labels) lose the location information of key targets during the abstraction process, and the fusion method of patch local features and CLS global features lacks adaptability, making it difficult to fully explore their complementarity. Ultimately, this results in semantic bias, missing details, and low matching degree with the actual scene of the image in the generated description. Although existing methods and technologies have achieved good results in the description of natural images, these methods have design flaws in areas such as local feature enhancement, global-local feature fusion, and location information preservation, which cannot further improve the performance of image description models. Therefore, it is urgent to design a feature processing and fusion architecture for complex image characteristics, so as to enhance local structural features, supplement the location information of global features, and achieve adaptive collaborative modeling of global and local features, thereby improving the accuracy and semantic discriminability of image text description. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide an image text description generation method and system based on anchor point visual feature diffusion, which addresses the above-mentioned problems in existing image text description methods. This invention aims to solve the technical problems of weak local structural feature representation, loss of positional information of global features, and lack of adaptability in global-local feature fusion in existing image text description methods, thereby improving the accuracy, semantic richness, and scene matching degree of image text description.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: An image text description generation method based on anchor point visual feature diffusion includes the following steps: extracting global and local features from the input image respectively; reshaping and encoding the local features in two-dimensional space and then performing anchor point visual feature diffusion to obtain local enhanced features; performing cross-insertion operation between the global features and the original local features, and then using a bidirectional Mamba encoder to perform bidirectional context modeling to obtain global enhanced features; fusing the local enhanced features and the global enhanced features to generate a fused unified visual representation; and gradually decoding the unified visual representation to generate an image text description statement.

[0005] Optionally, the step of extracting global and local features from the input image includes: inputting the input image into a CLIP visual encoder, obtaining the CLS global token and patch local token output from the penultimate layer of the CLIP visual encoder, extracting global features from the CLS global token using a global adapter, and extracting local features from the patch local token using a local adapter.

[0006] Optionally, the function expression of the global adapter is: ; The function expression for the local adapter is: ; in, As a global feature, For global adapter, For CLS global token, Indicates along the sequence axis Copy once To modify the activation function of the linear unit, For layer normalization operation, and These are the learnable weights and bias parameters in the global adapter, respectively. For local features, For local adapters, For patch local tokens, This is a convolutional module with a kernel size of 3×3. and These are the learnable weights and bias parameters in the local adapter, respectively.

[0007] Optionally, the step of performing two-dimensional spatial reshaping and multi-directional encoding on local features, followed by anchor point visual feature diffusion to obtain local enhanced features includes: performing two-dimensional spatial reshaping on local features to obtain locally reshaped features, and then performing multi-directional encoding on the locally reshaped features according to the following formula to obtain multi-directional encoded features: ; in, For multi-directional encoding features, For layer normalization operation, and These are the learnable weights and bias parameters, respectively. For splicing operations, This represents the sequence processing of the k-th Mamba encoder. For the k-th rotation transformation, the multi-directional encoding includes sequentially performing rotation transformations in four clockwise directions: 0°, 90°, 180°, and 270°. For local reshaping features; the multi-directional encoded features are reshaped in two-dimensional space to obtain multi-directional encoded reshaped features, and the multi-directional encoded reshaped features are aggregated into anchor point features according to the following formula to obtain the anchor point feature map: ; in, For anchor point feature maps, For a convolutional module with a kernel size of 1×1, Let Gaussian error be the activation function of the linear unit. This is a convolutional module with a kernel size of 3×3 and a stride of 2. To reshape features for multi-directional encoding; the anchor feature map is backfilled with anchor points according to the following formula to obtain the backfilled features: ; in, Features after backfilling At the anchor point The characteristics of the place, To balance the learnable parameters of the original features and the enhanced features, Reconstructing features for multi-directional encoding At the anchor point The characteristics of the place, The weight matrix for anchor point backfilling. Indicates taking The Middle The projected feature vectors of each aggregation unit for Mapping to multi-directional encoding remodeling features anchor point position The flattened index; after backfilling, the anchor point diffusion is globally guided according to the following formula to obtain locally enhanced features: ; ; ; ; ; ; ; in, For local enhancement features, The non-anchor point features after diffusion, For the original anchor point features, The anchor point features after diffusion. For binary anchor mask, For diffusion weights, For feature weights, Features after backfilling It is a deep convolutional layer. For the enhanced original anchor point features, For anchor point gain, For diffusion weights, feature weights, or anchor point gains, For the Sigmoid activation function, and These are the learnable weights and bias parameters, respectively. Refine global features for anchor points. For average pooling operation, As a global feature, and These are the learnable weights and bias parameters, respectively. Features after backfilling In the Each aggregation unit corresponds to a feature at the anchor point location.

[0008] Optionally, the step of performing a cross-insertion operation between the global features and the original local features, and then using a bidirectional Mamba encoder for bidirectional context modeling to obtain global augmented features includes: performing a cross-insertion operation between the global features and the original local features to generate an interleaved sequence, wherein each patch local token in the interleaved sequence is followed by an identical CLS global token; inputting the interleaved sequence into two parallel Mamba encoders (forward and reverse), and summing the outputs of the two parallel Mamba encoders element by element to obtain a fused sequence; and extracting the features at the original CLS global token positions from the fused sequence as global augmented features.

[0009] Optionally, the process of fusing local and global enhancement features to generate a unified visual representation includes: performing layer normalization on the local enhancement features, then generating spatial gating weights for each patch's local tokens through a linear layer and sigmoid activation; then generating scale and offset coefficients for the local enhancement features through two independent linear layers; and adaptively modulating the global enhancement features based on the scale, offset, and spatial gating weights to obtain modulated features. ; in, Features after modulation For global feature enhancement, For spatial gating weights, For element-wise multiplication, The hyperbolic tangent activation function is used. The scaling factor. The offset coefficient is used; the modulated features are input into the Mamba encoder, and the features extracted by the Mamba encoder are normalized by the layers to obtain the fused unified visual representation.

[0010] Optionally, when progressively decoding the unified visual representation to generate image text descriptions, the process includes: performing word embedding encoding on the already generated description sequence and superimposing sinusoidal positional encoding to obtain an encoded sequence; fusing the encoded sequence with the unified visual representation through a cross-attention module, and obtaining cross-modal fusion features through a feedforward network and normalization. ; ; in, Features obtained by fusion of cross-attention modules For root mean square normalization, For cross-attention modules, For encoded sequences, To unify visual representation, It is a feedforward network with GELU activation. The cross-modal fusion features are input into the Mamba2 decoder. The hidden states obtained by decoding the Mamba2 decoder are then used to predict the probability distribution of the next token in the generated description sequence through linear projection and Softmax activation.

[0011] The present invention also provides an image text description generation system based on anchor point visual feature diffusion, comprising a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the image text description generation method based on anchor point visual feature diffusion.

[0012] The present invention also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the image text description generation method based on anchor point visual feature diffusion via a processor.

[0013] The present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the image text description generation method based on anchor point visual feature diffusion via a processor.

[0014] Compared with existing technologies, the present invention can mainly achieve the following beneficial effects: The image text description generation method based on anchor point visual feature diffusion of the present invention strengthens the spatial structure representation of local features through anchor point feature diffusion, supplements the positional details of global features, realizes adaptive collaborative modeling of global and local features, enhances the model's perception and understanding of image semantics, and thus generates more accurate image text descriptions. It can solve the technical problems of weak local structural feature representation, loss of positional information of global features, and lack of adaptability in global-local feature fusion in existing image text description methods, and improve the accuracy, semantic richness and scene matching of image text descriptions. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of the overall network structure in an embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram of the network structure of the anchor point visual feature diffusion module in an embodiment of the present invention.

[0018] Figure 4 This is a schematic diagram of the network structure of the space-gated FiLM-Mamba module in an embodiment of the present invention.

[0019] Figure 5 This is a schematic diagram of the network structure of the decoder in an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0021] like Figure 1 As shown, the image text description generation method based on anchor point visual feature diffusion in this embodiment includes the following steps: S101, extract global and local features from the input image respectively; S102: The local features are reshaped in two-dimensional space and encoded in multiple directions, and then the anchor visual feature diffusion is performed to obtain the local enhanced features; the global features are cross-inserted with the original local features, and then the bidirectional Mamba encoder is used to perform bidirectional context modeling to obtain the global enhanced features. S103, which fuses local enhancement features and global enhancement features to generate a unified visual representation after fusion; S104, decodes the unified visual representation step by step to generate image text description statements.

[0022] like Figure 2 As shown, step S101 in this embodiment is implemented using a CLIP visual encoder and an adapter. The adapter includes a global adapter and a local adapter, which respectively extract global features from the input image. Figure 2 (represented by C in the original text) and local features include: inputting the input image into the CLIP visual encoder, obtaining the CLS global token and patch local token from the penultimate layer output of the CLIP visual encoder, denoted as C... , Where n is the number of patch tokens, For feature dimensions; utilize the global adapter to apply the CLS global token. Adjust the global features to the required dimensions, and utilize the local adapter to patch the local tokens. Adjust the local features to the required dimensions.

[0023] In this embodiment, the global adapter The function expression is: ; The local adapter The function expression is: ; in, As a global feature, For global adapter, For CLS global token, Indicates along the sequence axis Second copy (replacing the dimension with) The feature is copied n times to obtain the dimension. (characteristics) To modify the activation function of the linear unit, For layer normalization operation, and These are the learnable weights and bias parameters in the global adapter, respectively. For local features, For local adapters, For patch local tokens, This is a convolutional module with a kernel size of 3×3. and These are the learnable weights and bias parameters in the local adapter, respectively.

[0024] like Figure 2As shown, in this embodiment, step S102, extracting local enhanced features, is achieved through a bidirectional Mamba global feature enhancement module. The bidirectional Mamba global feature enhancement module integrates the location information of local patches into the global CLS features through cross-insertion, and combines bidirectional Mamba encoding to capture bidirectional context, thus overcoming the deficiency of traditional global features in losing location information and achieving effective fusion of global semantic abstraction and local location details. Specifically, the bidirectional Mamba global feature enhancement module performs two-dimensional spatial reshaping and multi-directional encoding on local features, and then performs anchor point visual feature diffusion to obtain local enhanced features, including: S201, Multi-directional Mamba Encoding: Local features are reshaped in two-dimensional space to obtain reshaped local features. These reshaped local features are then encoded in multiple directions according to the following formula to obtain multi-directional encoded features: ; in, For multi-directional encoding features, For layer normalization operation, and These are the learnable weights and bias parameters, respectively. For splicing operations, This represents the sequence processing of the k-th Mamba encoder. For the k-th rotation transformation, the multi-directional encoding includes sequentially performing rotation transformations in four clockwise directions: 0°, 90°, 180°, and 270°. This is a local remodeling feature; S202, Anchor Feature Aggregation: Multi-directional encoded features are reshaped in two-dimensional space to obtain multi-directional encoded reshaped features. These multi-directional encoded reshaped features are then aggregated into anchor features according to the following formula to obtain an anchor feature map: ; in, For anchor point feature maps, For a convolutional module with a kernel size of 1×1, Let Gaussian error be the activation function of the linear unit. This is a convolutional module with a kernel size of 3×3 and a stride of 2. Reconstruct features for multi-directional encoding; S203, Anchor Point Backfilling: Backfill the anchor points using the anchor point feature map according to the following formula to obtain the backfilled features: ; in, Features after backfilling At the anchor point The characteristics of the place, To balance the learnable parameters of the original features and the enhanced features, Reconstructing features for multi-directional encoding At the anchor point The characteristics of the place, The weight matrix for anchor point backfilling. Indicates taking The Middle The projected feature vectors of each aggregation unit for Mapping to multi-directional encoding remodeling features anchor point position Flattened index; anchor point feature map Remodeling features with multi-directional encoding The patch features corresponding to the aggregation center are fused, and the edge structure information of the original feature map is preserved to obtain the backfilled features. Features after backfilling at non-anchor point locations Remodeling features with multi-directional encoding Maintain consistency; S204, Globally Guided Anchor Point Diffusion: After backfilling, the anchor point diffusion is performed globally according to the following formula to obtain locally enhanced features: ; ; ; ; ; ; ; in, For local enhancement features, The non-anchor point features after diffusion, For the original anchor point features, The anchor point features after diffusion. For binary anchor mask, For diffusion weights, For feature weights, Features after backfilling It is a deep convolutional layer. For the enhanced original anchor point features, For anchor point gain, For diffusion weights, feature weights, or anchor point gains, For the Sigmoid activation function, and These are the learnable weights and bias parameters, respectively. Refine global features for anchor points. For average pooling operation, As a global feature, and These are the learnable weights and bias parameters, respectively. Features after backfilling In the Each aggregation unit corresponds to a feature at the anchor point location. For example... Figure 3 As shown, the global guided anchor point diffusion process includes the diffusion of global features. and The anchor point feature fusion yields the refined global anchor point feature. Generate diffusion weights Feature weights and anchor point gain ; through binary anchor mask mask The non-anchor region, after being amplified by anchor point gain, undergoes spatial diffusion of anchor point features through depthwise convolution, and then merges with the non-anchor region. The fusion process ultimately yields enhanced local features. .

[0025] like Figure 2 As shown, in this embodiment, step S102 extracts the global enhanced features through the anchor point visual feature diffusion module. The anchor point visual feature diffusion module enhances the spatial layout perception of local features through multi-directional Mamba encoding. Combined with anchor point aggregation, backfilling, and globally guided diffusion operations, it effectively solves the problems of weak structural representation and fragmented local patch features in traditional methods, strengthening the cross-patch structural association and key target perception capabilities of local features. Specifically, the anchor point visual feature diffusion module performs a cross-insertion operation between global features and the original local features, and then uses a bidirectional Mamba encoder to perform bidirectional context modeling to obtain the global enhanced features, including: S301, Cross-insertion: A cross-insertion operation is performed between the global features and the original local features to generate an interleaved sequence. Each patch local token in the interleaved sequence is followed by the same CLS global token. The adapted global features are then... Compared with the original local features By performing index-level cross insertion, an interleaved sequence of length 2n can be obtained. ; S302, Bidirectional Mamba Encoding: The interleaved sequence is input into two parallel Mamba encoders, one forward and one reverse. The outputs of the two parallel Mamba encoders are summed element-wise to obtain the fused sequence, which can be represented as: ; ; ; in, For fusion sequences, and These are the forward and reverse Mamba encoders, respectively. This is a sequence reversal operation. It is an interleaved sequence; S303, Global Feature Extraction: Extract the features of the original CLS global token positions from the fused sequence as global enhancement features, which can be represented as: ; in, This is an operation to extract the characteristics of the original CLS global token location.

[0026] like Figure 2 and Figure 4 As shown, in step S103 of this embodiment, the fusion of local and global enhancement features to generate a unified visual representation is achieved through a spatially gated FiLM-Mamba module. The spatially gated FiLM-Mamba module adaptively allocates the contribution weights of local patches through a spatial gating mechanism, combines FiLM feature linear modulation to achieve refined adaptive modulation of global features, and then completes the collaborative modeling of global and local features through a Mamba encoder. This fully leverages the complementarity of the two features, improving the adaptability and effectiveness of feature fusion. Specifically, the spatially gated FiLM-Mamba module's fusion of local and global enhancement features to generate a unified visual representation includes: S401, Spatial Gating Weight Generation: Local enhancement features are layer-normalized, and then spatial gating weights for each patch's local tokens are generated using a linear layer and sigmoid activation. This can be expressed as: ; in, For spatial gating weights, It is the Sigmoid activation function. and For learnable weights and bias parameters, For layer normalization; S402, FiLM Feature Modulation: The local enhancement features are then processed through two independent linear layers to generate scale and offset coefficients, which can be expressed as: ; in, Scaling factor or offset coefficient , and For learnable weights and bias parameters, For local enhancement features, Scaling factor and offset coefficient The index; based on the scale coefficient, offset coefficient, and spatial gating weights, the global enhanced features are adaptively modulated to obtain the modulated features: ; in, Features after modulation For global feature enhancement, For spatial gating weights, For element-wise multiplication, The hyperbolic tangent activation function is used. The scaling factor. This is the offset coefficient; S403, Collaborative Modeling and Fusion: The modulated features are input into the Mamba encoder. The features extracted by the Mamba encoder are normalized by layers to obtain a unified visual representation after fusion, which can be represented as: ; in, To unify visual representation, For layer normalization, For Mamba encoder.

[0027] like Figure 2 and Figure 5 As shown, in this embodiment, step S104, which progressively decodes the unified visual representation to generate the image text description, is implemented through a decoder. The process of progressively decoding the unified visual representation to generate the image text description includes: S501, Text Sequence Encoding: Encoding the already generated description sequence Perform word embedding encoding (when generating the first word of the image text description statement, the already generated description sequence) Contains only one start character , ), and then superimpose sinusoidal positional codes to obtain the encoded sequence. , can be represented as: ; in, For word embedding layer, For sinusoidal position coding function, For location index ( Figure 5 (where P represents the position index). S502, Cross-modal cross-attention fusion: This involves integrating the encoded sequence... With unified visual representation Cross-modal fusion features are obtained through fusion via cross-attention modules, followed by a feedforward network and normalization: ; ; in, Features obtained by fusion of cross-attention modules For root mean square normalization, For cross-attention modules, For encoded sequences, To unify visual representation, It is a feedforward network with GELU activation. For cross-modal fusion features; such as Figure 5 As shown, the cross-attention module unifies visual representation. As key K and value V, to encode the sequence By querying Q, the encoded sequence can be obtained. With unified visual representation Cross-attention features ; S503 and Mamba2 autoregressive decoding: fusing features across modalities Input Mamba2 decoder Mamba2 decoder Hidden state obtained through decoding The probability distribution of the next token in the generated description sequence is predicted by linear projection and softmax activation. , can be represented as: ; ; in, For normalized exponential functions, and For learnable weights and bias parameters, The prediction token at time t, This represents all tokens before time t. In this embodiment, the method employs a cross-entropy loss function during the training phase to maximize the likelihood probability of the true sequence description. The loss function... The function expression is: ; in, This describes the length of the sequence.

[0028] To verify the effectiveness and advancement of the image text description generation method based on anchor point visual feature diffusion in this embodiment, experiments were conducted on the NWPU dataset, and five existing advanced algorithms were selected for comparison: ICNET, ASSMN, dual-prompt, HCNET, and mlca-net. The experiments used four commonly used metrics in the text description field (BLEU, METEOR, ROUGE, and CIDEr) to evaluate the performance of different methods. BLEU, an n-gram-based accuracy metric, progressively measures the lexical overlap and fluency of the generated text with the reference description, from single characters to four-character phrases, and includes four specific metrics from BLEU-1 to BLEU-4. METEOR, based on lexical matching, introduces synonyms and semantic associations, making it more aligned with human semantic understanding. ROUGE focuses on recall, paying attention to the coverage of the reference description by the generated text and the sentence structure similarity. CIDEr, specifically for image description scenarios, uses TF-IDF weighted n-gram matching to more accurately evaluate the overall semantic relevance of the sentence, and is one of the core evaluation metrics in this field. The final results are shown in Table 1.

[0029] Table 1. Comparison of experimental results of the method in this embodiment with other advanced algorithms on the NWPU dataset.

[0030] As shown in Table 1, the method in this embodiment outperforms other advanced algorithms on the NWPU dataset, demonstrating its effectiveness and advancement. It is evident that the anchor visual feature diffusion module in this embodiment enhances the spatial layout perception of local features through multi-directional Mamba encoding. Combined with anchor aggregation, backfilling, and globally guided diffusion operations, it effectively solves the problems of weak structural representation and fragmented local patch features in traditional methods, strengthening the cross-patch structural association and key target perception capabilities of local features. The bidirectional Mamba global feature enhancement module integrates the positional information of local patches into global CLS features through cross-insertion, and captures bidirectional context through bidirectional Mamba encoding, compensating for the loss of positional information in traditional global features, and achieving effective fusion of global semantic abstraction and local positional details. Spatial gating FiLM- The Mamba module adaptively allocates the contribution weights of local patches through a spatial gating mechanism, and combines FiLM feature linear modulation to achieve refined adaptive modulation of global features. Then, the Mamba encoder completes the collaborative modeling of global and local features, fully exploring the complementarity between the two and improving the adaptability and effectiveness of feature fusion. The overall architecture of the method in this embodiment is designed for the characteristics of image spatial layout being cluttered and scaled. The collaborative work of each module realizes local feature enhancement, global feature optimization, and adaptive fusion of global and local features. The final generated description is highly accurate, semantically rich, and has a good match with the actual scene of the image, making it suitable for automatic text description tasks of various types of images.

[0031] Furthermore, this embodiment also provides an image text description generation system based on anchor point visual feature diffusion, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the image text description generation method based on anchor point visual feature diffusion. This embodiment also provides a computer-readable storage medium storing a computer program or instructions programmed or configured to execute the image text description generation method based on anchor point visual feature diffusion via a processor. This embodiment also provides a computer program product, including a computer program or instructions programmed or configured to execute the image text description generation method based on anchor point visual feature diffusion via a processor.

[0032] Those skilled in the art will understand that the technical solutions provided by this invention may take the form of a method, system, or computer program product. Therefore, this invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this invention may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce an implementation of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0033] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for generating image text descriptions based on anchor point visual feature diffusion, characterized in that, The process includes the following steps: extracting global and local features from the input image; reshaping and encoding the local features in two dimensions and then performing anchor visual feature diffusion to obtain enhanced local features; and performing cross-insertion operations between the global features and the original local features, followed by bidirectional context modeling using a bidirectional Mamba encoder to obtain enhanced global features. The local and global enhancement features are fused to generate a unified visual representation; the unified visual representation is then decoded step by step to generate image text descriptions.

2. The image text description generation method based on anchor point visual feature diffusion according to claim 1, characterized in that, The steps of extracting global and local features from the input image include: inputting the input image into the CLIP visual encoder, obtaining the CLS global token and patch local token output from the penultimate layer of the CLIP visual encoder, extracting global features from the CLS global token using a global adapter, and extracting local features from the patch local token using a local adapter.

3. The image text description generation method based on anchor point visual feature diffusion according to claim 2, characterized in that, The function expression for the global adapter is: ； The function expression for the local adapter is: ； in, As a global feature, For global adapter, For CLS global token, Indicates along the sequence axis Copy once To modify the activation function of the linear unit, For layer normalization operation, and These are the learnable weights and bias parameters in the global adapter, respectively. For local features, For local adapters, For patch local tokens, This is a convolutional module with a kernel size of 3×3. and These are the learnable weights and bias parameters in the local adapter, respectively.

4. The image text description generation method based on anchor point visual feature diffusion according to claim 1, characterized in that, The process of reshaping local features in two dimensions and encoding them in multiple directions, followed by anchor point visual feature diffusion to obtain enhanced local features, includes: reshaping local features in two dimensions to obtain reshaped local features, and encoding the reshaped local features in multiple directions according to the following formula to obtain multi-directional encoded features: ； in, For multi-directional encoding features, For layer normalization operation, and These are the learnable weights and bias parameters, respectively. For splicing operations, This represents the sequence processing of the k-th Mamba encoder. For the k-th rotation transformation, the multi-directional encoding includes sequentially performing rotation transformations in four clockwise directions: 0°, 90°, 180°, and 270°. For local reshaping features; the multi-directional encoded features are reshaped in two-dimensional space to obtain multi-directional encoded reshaped features, and the multi-directional encoded reshaped features are aggregated into anchor point features according to the following formula to obtain the anchor point feature map: ； in, For anchor point feature maps, For a convolutional module with a kernel size of 1×1, Let Gaussian error be the activation function of the linear unit. This is a convolutional module with a kernel size of 3×3 and a stride of 2. To reshape features for multi-directional encoding; the anchor feature map is backfilled with anchor points according to the following formula to obtain the backfilled features: ； in, Features after backfilling At the anchor point The characteristics of the place, To balance the learnable parameters of the original features and the enhanced features, Reconstructing features for multi-directional encoding At the anchor point The characteristics of the place, The weight matrix for anchor point backfilling. Indicates taking The Middle The projected feature vectors of each aggregation unit for Mapping to multi-directional encoding remodeling features anchor point position The flattened index; after backfilling, the anchor point diffusion is globally guided according to the following formula to obtain locally enhanced features: ； ； ； ； ； ； ； in, For local enhancement features, The non-anchor point features after diffusion, For the original anchor point features, The anchor point features after diffusion. For binary anchor mask, For diffusion weights, For feature weights, Features after backfilling It is a deep convolutional layer. For the enhanced original anchor point features, For anchor point gain, For diffusion weights, feature weights, or anchor point gains, For the Sigmoid activation function, and These are the learnable weights and bias parameters, respectively. Refine global features for anchor points. For average pooling operation, As a global feature, and These are the learnable weights and bias parameters, respectively. Features after backfilling In the Each aggregation unit corresponds to a feature at the anchor point location.

5. The image text description generation method based on anchor point visual feature diffusion according to claim 1, characterized in that, The step of performing a cross-insertion operation between global features and original local features, and then using a bidirectional Mamba encoder for bidirectional context modeling to obtain global augmented features includes: performing a cross-insertion operation between global features and original local features to generate an interleaved sequence, wherein each patch local token in the interleaved sequence is followed by an identical CLS global token; inputting the interleaved sequence into two parallel Mamba encoders (forward and reverse), and summing the outputs of the two parallel Mamba encoders element by element to obtain a fused sequence; and extracting the features at the original CLS global token positions from the fused sequence as global augmented features.

6. The image text description generation method based on anchor point visual feature diffusion according to claim 1, characterized in that, The process of fusing local and global enhancement features to generate a unified visual representation includes: performing layer normalization on the local enhancement features, then generating spatial gating weights for each patch's local tokens through a linear layer and sigmoid activation; then generating scale and offset coefficients for the local enhancement features through two independent linear layers; and finally, adaptively modulating the global enhancement features based on the scale, offset, and spatial gating weights to obtain the modulated features. ； in, Features after modulation For global feature enhancement, For spatial gating weights, For element-wise multiplication, The hyperbolic tangent activation function is used. The scaling factor. The offset coefficient is used; the modulated features are input into the Mamba encoder, and the features extracted by the Mamba encoder are normalized by the layers to obtain the fused unified visual representation.

7. The image text description generation method based on anchor point visual feature diffusion according to claim 1, characterized in that, The process of progressively decoding the unified visual representation to generate image text descriptions includes: encoding the generated description sequence with word embeddings and superimposing sinusoidal positional encodings to obtain an encoded sequence; fusing the encoded sequence with the unified visual representation through a cross-attention module, and then obtaining cross-modal fusion features through a feedforward network and normalization. ； ； in, Features obtained by fusion of cross-attention modules For root mean square normalization, For cross-attention modules, For encoded sequences, To unify visual representation, It is a feedforward network with GELU activation. The cross-modal fusion features are input into the Mamba2 decoder. The hidden states obtained by decoding the Mamba2 decoder are then used to predict the probability distribution of the next token in the generated description sequence through linear projection and Softmax activation.

8. An image text description generation system based on anchor point visual feature diffusion, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the image text description generation method based on anchor point visual feature diffusion as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the image text description generation method based on anchor point visual feature diffusion as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the image text description generation method based on anchor point visual feature diffusion as described in any one of claims 1 to 7.

Images