Robust image caption generation method based on fine-grained noise perception

Abstract

The application discloses a kind of robust image caption generation methods based on fine-grained noise perception, comprising: collecting multimodal graph-text pair training data;Extract the grid feature and global feature of image and obtain refined grid feature and refined global feature by processing, extract text feature;Pre-fuse image refined global feature and text feature, model the correlation of image refined grid feature and the graph-text pre-fusion feature;Conversion processing is carried out to the interaction result to obtain first-order word prediction;With first-order word prediction vector instead of corresponding label word vector as input, obtain second-order word prediction, quantize the cumulative error of second-order word prediction and word label;According to the quantization result, construct pseudo-label by weighted fusion of word label and model prediction, re-cross-modal interaction is carried out, and image caption is output.The application provides the identification mode of fine-grained data weak correlation noise, and is gradually corrected in model fitting process, effectively improves the performance of image caption model in data weak correlation noise scene.

Description

Technical Field

[0001] This invention relates to the field of image captioning technology, and in particular to a robust image captioning generation method based on fine-grained noise perception. Background Technology

[0002] In the real world, people perceive information in various ways, making multimodal learning a constant research focus in the field of artificial intelligence. Image captioning generation is a crucial area of ​​research in multimodal learning, aiming to enable computers to describe given image content using natural language. It has wide applications in various life scenarios, including e-commerce and assistive technology for the visually impaired.

[0003] Its core task is to learn the linguistic feature representation of visual features. Common architectures include using convolutional neural networks (CNNs) as image encoders and long short-term memory networks (LSTMs) as text decoders. To better consider the local and global features of images, some studies perform image region segmentation and introduce attention mechanisms to model the correlation information between regions. In recent years, the development of multi-head self-attention models has further improved the modeling of intra-modal and inter-modal relationships, significantly improving the quality of generated captions. However, high-performance image captioning models usually require large-scale image-text pairs for training, but this data is often collected from the Internet and is prone to noise problems, i.e., the image-text pairs are not perfectly matched. The interference of mismatched noise will mislead the model's fitting, resulting in a large deviation between the model-generated captions and the actual image content. Existing directions attempt to weight image-text pairs to reduce the impact of noise, but ignore the case of partial image-text matching, failing to maximize the use of available information in the text labels. Therefore, how to maximize the use of known image-text pair information while maintaining the robustness and noise resistance of the model has become an urgent technical problem to be studied. Summary of the Invention

[0004] To address the problem of fine-grained processing of noisy image-text pairs in image captioning tasks under weak correlation of training data, this invention proposes a robust image captioning generation method based on fine-grained noise perception. This method finely assesses the reliability of each word in the text annotation, performs word-level reweighting to determine whether to adequately fit the pairing of the word with the visual region, and corrects noisy words during the model fitting process. This maximizes the utilization of available information in the text annotation, thereby significantly improving the model's learning performance on noisy image-text pairs.

[0005] The technical solution adopted in this invention is as follows: A robust image caption generation method based on fine-grained noise perception, comprising the following steps:

[0006] Step 1: Collect multimodal image-text pair training data;

[0007] Step 2: Extract the grid features and global features of the image and process them to obtain refined grid features and refined global features, and extract text features;

[0008] Step 3: Refine the global features and text features of the pre-fused image, model the correlation between the image refinement grid features and the image-text pre-fused features, and realize cross-modal interaction;

[0009] Step 4: Transform the interaction results to obtain first-order word predictions and quantify the prediction determinism;

[0010] Step 5: Replace the corresponding label word vector with the first-order word prediction vector as input to obtain the second-order word prediction and quantify the cumulative error between the second-order word prediction and the word label.

[0011] Step 6: Based on the quantization results, weighted fusion of word tags and model predictions is used to construct pseudo-tags, cross-modal interaction is performed again, and image captions are output.

[0012] In step 2, the original image is further input into the encoder for feature extraction to obtain image grid features and global features, including:

[0013] A window self-attention transformer is used as an encoder to extract mesh features;

[0014] The average pooling of the grid features is calculated as a global feature.

[0015] Further, the image mesh features and global features are input into a thinning encoder to extract intramodal correlation information, obtaining thinned mesh features and thinned global features, including:

[0016] Based on the window self-attention mechanism, the relationship between features within a modality is captured, and image features carrying information about intramodal correlation are obtained. That is, when multi-head self-attention is calculated in each window, global features are additionally concatenated to the key vector K and the value vector V.

[0017] The image features carrying intramodal correlation information are input into the feedforward neural network and then subjected to layer normalization processing. The final refined grid features and refined global features are obtained from the last layer block.

[0018] Furthermore, the original text is input into the encoder for feature extraction to obtain the text-encoded semantic features.

[0019] In step 3, the image refinement global features are further fused into the text features to be input to the decoder, including:

[0020] The semantic features of the text encoding are concatenated with the global features of the image refinement. The concatenated features are then processed using an activation function and subjected to layer normalization.

[0021] Language masking modeling is performed on the processed spliced ​​features to obtain pre-fused image-text features carrying interactive information within the text modality.

[0022] Furthermore, the modal interaction correlation information between the image refinement mesh features modeled based on cross-multi-head self-attention and the image-text pre-fusion features includes:

[0023] The image-text fusion features are used as query vectors, and the image refinement grid features are used as key and value vectors. These are then input into the decoder for interactive processing to obtain cross-modal interaction information.

[0024] In step 4, the interaction result is further transformed to obtain first-order word predictions, and the prediction confidence is quantified, including:

[0025] The interaction results are linearly transformed, and an activation function is used to map the category with the highest posterior probability to obtain first-order word predictions.

[0026] The entropy of the first-order word prediction is calculated to quantify the certainty of the model prediction, and its value is normalized.

[0027] In step 5, the first-order word prediction vector is used instead of the corresponding label word vector as input to obtain the second-order word prediction. The cumulative error between the second-order word prediction and the word label is quantified, including:

[0028] The first-order word prediction vector is used to replace the corresponding label word vector and input into the decoder. The second-order word prediction sequence is obtained through cross-modal interaction.

[0029] The cross-entropy loss value between the second-order word prediction sequence and the word tag sequence is then used to quantify the cumulative error of generating second-order predictions based on the first-order prediction sequence, and its value is normalized.

[0030] In step 6, further, based on the quantization results, the word tags and model predictions are weighted and fused to construct pseudo-tags, and cross-modal interaction is performed again to output image captions, including:

[0031] The product factor is calculated based on the model's convergence curve, and the sum of the quantized values ​​is mapped to [0,1], serving as the word reliability weight ∈ l ;

[0032] ∈ l As the weight of the original word label, 1-∈ l As weights for word prediction in the model, the word labels are weighted and fused with the model predictions to construct pseudo-labels;

[0033] Use pseudo-tags to re-perform cross-modal interaction calculations;

[0034] The interaction results are linearly transformed, and an activation function is used to map the category with the highest posterior probability to obtain a word prediction sequence, which is then output as an image caption.

[0035] Compared with existing technologies, the significant advantages of this invention are as follows: The robust image captioning generation method based on fine-grained noise awareness provided by this invention adds a review process for the reliability of each word tag to the classic cross-modal interaction paradigm. Specifically, the model measures the reliability of each word tag from two aspects: inter-modal representativeness and intra-modal discriminative power, to identify mismatched words in a fine-grained manner. The former quantifies the correlation between word tag embedding features and image features through the deterministic prediction of the model, while the latter amplifies the cumulative error of captioning generation by obtaining second-order predictions through secondary decoding of model predictions. During the optimization process, the model determines whether to fully fit the region-word pair based on the reliability of the current word tag, and gradually fuses word tags with model word predictions to construct pseudo-labels for supervised cross-modal interaction learning as the model converges. This robust image captioning generation method based on fine-grained noise awareness can effectively alleviate the interference of weakly correlated noise in data on model training by finely identifying and processing word noise, while making full use of the correct relevant information of partially matched image-text pairs, thus ultimately significantly improving the performance of image captioning generation. Attached Figure Description

[0036] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the following detailed description to explain the invention, but do not constitute a limitation thereof.

[0037] Figure 1 The overall flowchart of the robust image caption generation method based on fine-grained noise perception provided by the present invention is shown.

[0038] Figure 2 This is a schematic diagram of the robust image caption generation method based on fine-grained noise perception provided by the present invention.

[0039] Figure 3 The flowchart for processing graphic pairs provided by this invention.

[0040] Figure 4 The image feature extraction flowchart provided by this invention.

[0041] Figure 5 The flowchart for image feature refinement processing provided by this invention.

[0042] Figure 6 The cross-modal interaction flowchart provided by this invention.

[0043] Figure 7 The flowchart for image and text feature pre-fusion provided by the present invention.

[0044] Figure 8 The image caption generation example is shown in the comparison diagram. Detailed Implementation

[0045] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The specific implementation of the present invention will be described below with reference to the accompanying drawings.

[0046] To better convey the inventive solution, the technical solution has been clearly and completely described with reference to the accompanying drawings of the embodiments. Obviously, the embodiments described represent only a part of the present invention, and not all of it. Without inventive effort, those skilled in the art can obtain other embodiments falling within the scope of protection of the present invention based on these embodiments.

[0047] It should be emphasized that the use of "comprising" and "having" and their variations is intended to cover a non-exclusive scope, that is, a process, method, system, product or device that includes a series of steps or units, not limited to the steps or units explicitly listed, but may also include other steps or units not explicitly listed or inherent to these processes, methods, products or devices.

[0048] This embodiment provides a robust image caption generation method based on fine-grained noise perception. Figure 1 This is an overall flowchart of a robust image caption generation method based on fine-grained noise perception provided in an embodiment of the present invention. Figure 2 This is a corresponding framework diagram. For example... Figure 1 , 2 As shown, it includes:

[0049] S100. Collect multimodal image-text pair data;

[0050] In this embodiment of the invention, a large number of image-text pairs are obtained from a publicly available image caption dataset for model training and testing.

[0051] S200. Extract the grid features and global features of the image and process them to obtain refined grid features and refined global features, and extract text features;

[0052] The image-text pairs obtained above will be input into the encoder for image feature extraction and text feature extraction, respectively.

[0053] Specifically, for images, a window self-attention transformer is used as an encoder to extract grid features V. G And calculate the average pooling of the grid features as the global feature v. gThis process yields image mesh features and global features. Then, by capturing the relationships between intra-modal features using a window-based self-attention mechanism, the image mesh features and global features are input into a thinning encoder to extract intra-modal correlation information, thereby obtaining image features carrying intra-modal correlation information. Next, the image features carrying intramodal correlation information are input into the feedforward neural network and subjected to layer normalization processing. The final refined grid features and refined global features are obtained from the last layer block. and Compared to using a two-stage object detection model for region feature extraction, the window self-attention transformer offers superior computational efficiency and focuses on reflecting local structure and details, making it more suitable for fine-grained examination of visual region matching with word labels. The refinement encoder further captures the relationships between image regions, enhancing the representation learning ability of intramodal correlation information and contributing to a better understanding of the visual structure of image captioning models.

[0054] For text, feature extraction is relatively simple. The original text is directly input into the encoder to extract features and obtain the text-encoded semantic features.

[0055] S300: Pre-fused image refinement global features and text features, modeling the correlation between image refinement grid features and the image-text pre-fused features, to achieve cross-modal interaction;

[0056] The semantic features of the text encoding are concatenated with the global features of the image refinement. An activation function is used to process the concatenated features, followed by layer normalization, to fuse the global features of the image refinement into the text features to be input into the decoder. Then, language masking is performed on the processed concatenated features to obtain pre-fused image-text features carrying text modality interaction information. Using only image grid features can easily limit the model's inference ability due to insufficient global context information. In contrast, the integration of global image features can be seen as a warm-up for cross-modal interaction, compensating for and promoting the model's overall perception of image-text pair matching. Next, formal cross-modal interaction learning is performed, namely, modeling the modal interaction relevance information between the image refinement grid features and the pre-fused image-text features based on cross-multi-head self-attention. The pre-fused image-text features are used as query vectors, and the image refinement grid features are used as key and value vectors, respectively, and input together into the decoder for interaction processing to obtain cross-modal interaction information.

[0057] S400: The interaction result is transformed to obtain first-order word prediction and the prediction determinism is quantified.

[0058] It should be understood that the quantification of prediction certainty assesses the mutual expressiveness of visual language modalities from the perspective of intermodal representativeness. Intuitively, the content words in the caption will correspond to at least one visual region in the relevant image. Therefore, based on the corresponding cross-modal attention and contextual information, the decoder will give a more confident word prediction when visual region-word matching is achieved, and a less certain prediction otherwise, i.e., the entropy of the model prediction has a larger ground truth value.

[0059] Specifically, the interaction result is linearly transformed, and an activation function is used to map the category with the highest posterior probability to obtain a first-order word prediction, which takes the following form:

[0060]

[0061] Among them W x It is a linear transformation matrix with learnable parameters. For cross-modal interaction information, p(x) t |x 1:t-1 p represents the model's prediction of the word at time t based on the prefix word sequence generated at time t-1. t .

[0062] Then, the entropy of the first-order word prediction is calculated to quantify the certainty of the model prediction, and its value is normalized, specifically in the following form:

[0063] ∈ t,2 =1-Norm(H(p) t+1 ))

[0064] S500: Use the first-order word prediction vector instead of the corresponding label word vector as input to obtain the second-order word prediction and quantify the cumulative error between the second-order word prediction and the word label.

[0065] It should be understood that quantifying the cumulative error of second-order word prediction evaluates the quality of text generated from the model's predicted word sequence from the perspective of intramodal discriminative power. Since image caption generation exhibits the characteristic of cumulative error, meaning the quality of the leading prefix largely determines the quality of subsequent word generation, this step amplifies the discriminative power of the quantified prediction quality values ​​between matched words and noise words based on prediction, making it easier to examine the reliability of the target word.

[0066] Specifically, the first-order word prediction vector is used instead of the corresponding label word vector and input into the decoder. The second-order word prediction sequence is obtained through cross-modal interaction, in the following form:

[0067]

[0068]

[0069] Subsequently, the cross-entropy loss value between the second-order word prediction sequence and the word label sequence is used to quantify the cumulative error of generating second-order predictions based on the first-order prediction sequence, and its value is normalized, specifically in the following form:

[0070]

[0071] Where L XE p represents the cross-entropy loss. t+2:T ' is the second-order word prediction sequence, y t+2:T It is the corresponding word tag sequence.

[0072] S600. Based on the quantization results, the word tags and model predictions are weighted and fused to construct pseudo-tags, and cross-modal interaction is performed again to output image captions.

[0073] It should be understood that, based on the newly constructed pseudo-labels, the cross-modal interaction operation will be re-executed to obtain more reasonable interaction information that has, to some extent, eliminated weak data association noise. By performing a linear transformation on the interaction results and using an activation function to map the category with the highest posterior probability, the model obtains a corrected word prediction sequence, which is considered the final image caption output.

[0074] Specifically, the product factor is calculated based on the model's convergence curve, placing greater trust in the original labels in the early stages of training and greater trust in the model's predictions in the later stages. The sum of the quantized values ​​is then mapped to [0,1] as the word reliability weight ∈ t The specific form is as follows:

[0075]

[0076] ∈ t =h(γ)*(∈ t,1 +∈ t,2 )

[0077] Where γ and Γ are the current iteration number and the total iteration number, respectively, and C and τ are both hyperparameters. In each training round, the product factor h(γ) is adaptively adjusted based on the curvature of the loss curve of the previous round.

[0078] Then ∈ t As the weight of the original word label, 1-∈ t As weights for word prediction in the model, the weighted fusion of word tags and model predictions can construct fine-grained pseudo-tags at the word level, specifically in the following form:

[0079]

[0080] The robust image captioning generation method based on fine-grained noise perception provided in this invention introduces a review process for the reliability of each word tag on top of the traditional cross-modal interaction framework. This allows for the evaluation of the reliability of each word tag from two aspects: inter-modal representativeness and intra-modal discriminability, enabling fine-grained detection of mismatched words. Specifically, the former quantifies the correlation between word tag embedding features and image features through deterministic model prediction, while the latter amplifies the cumulative error of captioning generation by obtaining second-order predictions through secondary decoding of model predictions. During optimization, the model determines whether to adequately fit the region-word pair based on the reliability of the current word tag, gradually fusing word tags with model word predictions to construct pseudo-tags for supervised cross-modal interaction learning. Because this fine-grained noise perception-based method can precisely identify and process word noise, it can effectively reduce the interference of weakly correlated noise in the data on model training and fully utilize the correct relevant information of partially matched image-text pairs, thereby significantly improving the performance of image captioning generation.

[0081] In this embodiment of the invention, the grid features and global features of the image are extracted and processed to obtain refined grid features and refined global features. Text features are then extracted, which may specifically include:

[0082] The image to be generated is subjected to vector processing and preliminary feature extraction to obtain preliminary image features;

[0083] Intramodal correlation information is extracted from the preliminary image features to obtain refined grid features and refined global features;

[0084] Feature extraction is performed on the original text labels to obtain the text-encoded semantic features.

[0085] For example, such as Figure 3 As shown, the grid features and global features of the image are extracted and processed to obtain refined grid features and refined global features. Text features are extracted, including:

[0086] S210. Input the original image into the encoder for feature extraction to obtain image grid features and global features;

[0087] More specifically, such as Figure 4 As shown, the original image is input into the encoder for feature extraction to obtain image grid features and global features, including:

[0088] S211. Using a window self-attention transformer as an encoder to extract mesh features V G ;

[0089] It should be understood that the advantage of using this base model to extract grid features is that it can model global contextual information by leveraging the advantages of the self-attention mechanism, perceive the spatial position and visual structure of objects, and the sliding window mechanism can enable the model to adaptively handle the scale differences of a given input, making it less likely to miss the recognition of small objects.

[0090] S212. Calculate the average pooling of the grid features as the global feature v. g .

[0091] It should be understood that average pooling has a certain degree of position invariance. Performing this operation on the entire grid feature allows the model to capture features that are generally present in the image, rather than just features of specific locations or objects. Global image features are more generalizable, serving as a warm-up for the interaction of the visual language model and compensating for and promoting the model's overall perception of image-text pair matching.

[0092] S220. Input the image mesh features and global features into the thinning encoder to extract intramodal correlation information and obtain the thinned mesh features and the thinned global features;

[0093] More specifically, such as Figure 5 As shown, the image mesh features and global features are input into a thinning encoder to extract intramodal correlation information, obtaining thinned mesh features and thinned global features, including:

[0094] S221. Capture the relationship between features within a modality based on the window self-attention mechanism, and obtain image features carrying information about intramodal correlation.

[0095] During the multi-head self-attention computation in each window, global features are additionally concatenated to the key vector K and value vector V to obtain preliminary interaction results carrying intramodal correlation information. The specific format is as follows:

[0096]

[0097]

[0098] S222. The image features carrying intramodal correlation information are input into the feedforward neural network and then subjected to layer normalization processing to obtain refined grid features and refined global features.

[0099] For an N-block encoding module, after multiple layers of feature processing, the final refined grid features and refined global features are obtained from the output of the last block. and The specific form of the operation in each block l is as follows:

[0100]

[0101]

[0102] in, and These represent the mesh features and global features output by the previous block l-1, respectively, and are used as the input to the current block l. It is a linear transformation matrix with learnable parameters.

[0103] S230. Input the original text into the encoder for feature extraction to obtain the text encoding semantic features.

[0104] It should be understood that, in the embodiments of the present invention, given image description text, the image description text is vectorized to obtain image description text vector y, which is used as the input of the encoder.

[0105] After completing the feature extraction and processing of the image and text, a cross-modal interactive learning process will be performed. In this embodiment of the invention, the global features of image refinement and text features are pre-fused, and the correlation between the image refinement grid features and the pre-fused image-text features is modeled to achieve cross-modal interaction, including:

[0106] The image refinement global features are fused into the text features to be input to the decoder;

[0107] Modal interaction correlation information between image refinement mesh features and image-text pre-fusion features is modeled based on cross-multi-head self-attention.

[0108] For example, such as Figure 6 As shown, the pre-fused image refines global features and text features, models the correlation between the refined image grid features and the pre-fused image-text features, and realizes cross-modal interaction, including:

[0109] S310. Fuse the image's refined global features into the text features to be input to the decoder;

[0110] It should be understood that incorporating global features of an image into the model can be seen as a preprocessing step for cross-modal interaction, which helps to compensate for and enhance the model's ability to perceive the overall matching of image-text pairs.

[0111] More specifically, such as Figure 7 As shown, the image refinement global features are fused into the text features to be input to the decoder, including:

[0112] S311. Concatenate text encoding semantic features and image refinement global features, use activation functions to process the concatenated features and perform layer normalization;

[0113] The specific form is as follows:

[0114]

[0115] in W represents the text features output by the previous block l-1 at time t, and is used as the input to the current block l. f It is a linear transformation matrix with learnable parameters, and ReLU is the activation function.

[0116] S312. Perform language mask modeling on the processed splicing features to obtain pre-fused image-text features carrying text modal interaction information.

[0117] It should be understood that the language masking modeling here extracts contextual relevance information from the spliced ​​features and performs deeper feature fusion.

[0118] The specific form is as follows:

[0119]

[0120] in It is a linear transformation matrix with learnable parameters. It is the sequence of prefix words generated at time t-1.

[0121] S320. Modal interaction correlation information between image refinement mesh features and image-text pre-fusion features based on cross-multi-head self-attention modeling;

[0122] It should be understood that this process adopts the classic two-stage learning paradigm of image captioning tasks, namely the training (cross-entropy) and fine-tuning (reinforcement learning) stages, focusing on training the model's consistent understanding and expression of semantic features of images and text.

[0123] The image-text fusion features are used as the query vector, and the image refinement grid features are used as the key and value vectors, respectively. These are then input into the decoder for interactive processing to obtain cross-modal interaction information. The specific form of the operation in each block l is as follows:

[0124]

[0125]

[0126] in It is a linear transformation matrix with learnable parameters. As a feature of image and text fusion, It is an image refinement grid feature.

[0127] It should be noted that the robust image captioning generation method based on fine-grained noise perception in the embodiments of the present invention relies on the collaborative action of the encoder and decoder. During the training phase of the encoder and decoder models, a loss function must be calculated on the decoder output to reduce the gap between the predicted and true values, thereby accurately adjusting the parameter model. Subsequently, during the model inference phase, the trained model can be directly applied to generate image captions.

[0128] The calculation of the loss function mainly includes the following:

[0129] 1) Calculation of cross-entropy loss function.

[0130] Given visual features and a known prefix text sequence, the model will provide a current word generation prediction. Cross-entropy constrains the similarity between the model's word prediction and the given original label, and is used in the first stage of training, specifically in the following form:

[0131]

[0132] Where y 1:T The text feature sequence representing the description of the given image, i.e., the training labels. The network parameters θ define the policy p. θ .

[0133] 2) Calculation of the reinforcement learning loss function.

[0134] It should be understood that image captioning focuses more on generation quality during inference, while the calculation of metrics for measuring text quality is non-differentiable. Therefore, optimizing for classification loss during training leads to a mismatch between training and testing metrics. Reinforcement learning introduces a reward signal, treating the decoder as the agent and calculating the core metric CIDER (the quality index of generated image captions) as the reward. This allows the model to directly optimize the metric for measuring text quality through backpropagation, as shown in the following form:

[0135]

[0136] Where r(·) is the CIDER index score, and E is the expected value.

[0137] In summary, the robust image captioning generation method based on fine-grained noise perception provided by this invention explores noisy image captions in a fine-grained manner, carefully measures the reliability of word labels, and considers both inter-modal and intra-modal influences to identify mismatched words. Furthermore, a noise-resistant training strategy is designed to progressively weight and fuse model predictions with original labels to construct pseudo-word labels, thereby adaptively coordinating mismatched words, improving model robustness, and enhancing the quality of captioning generation.

[0138] This invention also provides a specific embodiment of a robust image caption generation method based on fine-grained noise perception, which can be implemented using a computer device. The specific processing flow of this method is as follows:

[0139] First, collect image-text pairs that can be used to train the image captioning generation model, or use publicly available classic datasets, and divide them into training data, validation data, and test data. The training data is used to guide the model to fit the data, the validation data is used to select the best model, and the test data is used to calculate the evaluation index of image captioning generation quality to verify the model performance.

[0140] Secondly, during the training phase, the images in the image-text pair are first processed by a window self-attention transformer network to extract grid features and global features. Specifically, each image is divided into 12*12 grid features, all of which are 512-dimensional. The global features obtained after average pooling are also 512-dimensional. Then, a refinement encoder is used to model contextual relationships to obtain refined grid features and global features, while keeping the dimensions unchanged. The text undergoes uniform lowercase writing, vocabulary construction, word frequency statistics, integer encoding, and padding to a uniform length of 17. The text semantic features are then encoded using a standard encoder. Subsequently, the visual features and text features are entered into the decoder to perform cross-modal interaction to generate a prediction sequence. Each word label is reweighted through a review process, and the loss function is calculated and backpropagation is performed based on pseudo-labels to optimize the consistency between the prediction sequence and the label sequence, and the model parameters are iteratively updated.

[0141] Finally, in the inference phase, the image to be used for subtitle generation follows the same processing method as in the training phase, obtaining refined grid features and global features before feeding them into the decoder. Unlike the training phase, the model does not use additional text feature vectors as input guide words for the decoder. Instead, it generates subtitles word by word starting from the start character. Subsequent prediction sequences are based solely on the start character and prefix predictions. The final output consists of a descriptive text generated for the entire image, along with scores from metrics such as CIDER and SPICE to evaluate the generation quality.

[0142] Example: Training and testing on the MSCOCO dataset

[0143] (1) Dataset: The experiment uses the classic public dataset MS-COCO, which contains 123,287 images, each with 5 captions. Based on the "Karpaty" segmentation method, this example uses 5,000 images as the validation set, 5,000 images as the test set, and the rest as the training set. Considering that the captions in the MS-COCO dataset are entirely manually annotated, this example actively constructs asymmetric noise samples to better simulate noisy scenes in the real world. Specifically, this means selecting interchangeable captions based on similar objects. For example, when two images contain the same object, their captions will be swapped to construct partially matching samples.

[0144] (2) Comparison Method: In this example experiment, a leading image captioning model in recent years was selected as the comparison method. AoANet added an attention-attention module to refine the attention results; M... 2 T designed a grid connection between the decoder and encoder, using priority knowledge by adding memory vectors in the attention module; RSTNet incorporates an adaptive attention module in the decoder, which can adaptively evaluate the contribution of visual and non-visual information before word prediction; DLCT considers both region features and grid features to obtain supplementary fine-grained details and contextual information; PureT uses a window self-attention transformer instead of an object detector to build an end-to-end image and text generation paradigm.

[0145] (2) Experimental Details: The embodiments of this invention are implemented based on the PyTorch deep learning framework, using the Python 3.6.11 compiler. Both the encoder and decoder models have 3 blocks, and the number of heads in the multi-head self-attention operation is set to 8. Real-time two-stage training is performed: the first stage uses cross-entropy loss for 25 rounds, and the second stage uses reinforcement learning for 30 rounds. The batch size is set to 10, and the initial learning rate is set to 5 × 10⁻⁶. -6 The optimizer used was ADAM, with a total of approximately 283,000 iterations. Hyperparameters C and τ were set to 0.5 and 10, respectively. The training device was an NVIDIA TITAN X graphics card. During training, the model was tested on a validation set every 12,000 iterations, and the model with the best performance on the validation set was saved.

[0146] (3) Experimental Results: This example uses an 80% noise ratio as an example for experimental analysis. Table 1 provides a quantitative comparison of the image captioning performance of the present invention (Ours) under the experimental settings with the comparison method and self-ablation. Evaluation metrics selected include BLEU@N, METEOR, ROUGE-L, CIDEr, and SPICE, abbreviated as "B@N", "M", "R", "C", and "S", respectively. The best performance results are shown in bold. As can be seen from the table, the example method implemented by the present invention achieves the following scores under the experimental settings: BLEU-1 score is 65.2, BLEU-4 score is 20.1, METEOR score is 23.4, ROUGE score is 45.8, CIDER score is 65.2, and SPICE score is 13.3. It achieves the best performance across all metrics, especially significantly exceeding the comparison method by 5 points or more in the core image captioning metric CIDER. This fully demonstrates the superiority of the robust image captioning generation method based on fine-grained noise perception provided by the present invention, which can effectively adaptively mitigate the adverse effects of noisy training data on model learning. Furthermore, this experiment supplements the investigation of the performance of variant methods for ablation several key modules in the examples of this invention, namely, removing the predictive deterministic quantization (Ours w / oε1), the second-order word prediction cumulative error quantization (Ours w / oε2), and the model convergence curve product factor (Ours w / oε(γ)). The lower half of the table reports the corresponding quantitative results. These variant methods still outperform the comparative methods, but show a performance decline compared to the experimental examples of this invention, which strongly demonstrates the effectiveness of the corresponding key modules. The predictive deterministic quantization of word prediction approaches the problem from an intermodal perspective, converting the cross-modal correlation information of visual language extracted from cross-modal interactions into entropy calculations for each word prediction. This allows for more intuitive and fine-grained discrimination of each word, while also being computationally convenient and efficient. The second-order word prediction cumulative error quantization approaches the problem from an intramodal perspective, focusing on the characteristics of text sequence generation. This allows for the reasonable amplification of the slight differences between matching words and noise words, significantly increasing their discriminative power, and is an important foundation for noise word recognition and pseudo-label construction. Simultaneously, the model convergence curve product factor can adaptively and progressively construct pseudo-labels to guide model fitting. Based on the memory effect of neural networks, the model prioritizes fitting simple, clean data patterns in the early stages of training. The product factor ensures that the model can identify and selectively fit noisy information only after it has a certain generalization ability.

[0147] Table 1 compares image captioning performance on the MSCOCO dataset containing 80% asymmetric noise.

[0148] AoANet 59.5 15.8 18.1 41.8 57.4 10.0 <![CDATA[M 2 T]]> 57.9 15.4 16.5 38.9 55.9 9.0 RSTNet 61.3 16.5 18.8 42.8 58.7 10.9 DLCT 62.6 17.4 20.6 43.3 60.2 11.3 PureT 63.2 17.4 20.6 43.3 60.2 11.3 Ours 65.2 20.1 23.4 45.8 65.2 13.3 <![CDATA[Ours w / oε1]]> 65.0 19.8 23.1 45.5 65.0 13.1 <![CDATA[Ours w / oε2]]> 64.3 18.8 22.1 45.0 63.9 12.5 Ours w / oh(γ) 64.0 18.7 22.0 45.1 63.6 12.4

[0149] (4) Effect Demonstration: In the visualization comparison, the captions describing the image content are very accurate. The visualization clearly shows that the embodiments of the present invention can effectively describe the content in the image. In a large number of comparative algorithm experiments, the deviation in descriptive images is also significantly reduced, and the text description scene effect in different scenarios is significantly improved, without any overgeneralization. Figure 8 The descriptive text in the algorithm model shows a significant improvement in performance due to the attention mechanism applied to text features.

[0150] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robust image caption generation method based on fine-grained noise perception, characterized in that, Includes the following steps: Step 1: Collect multimodal image-text pair training data; Step 2: Extract the grid features and global features of the image and process them to obtain refined grid features and refined global features, and extract text features; Step 3: Refine global and text features of the pre-fused image, model the correlation between the refined image grid features and the pre-fused image-text features, and realize cross-modal interaction; Step 4: Transform the interaction results to obtain first-order word predictions and quantify the prediction certainty, including: The interaction results are linearly transformed, and an activation function is used to map the category with the highest posterior probability to obtain first-order word predictions. The entropy of the first-order word prediction is calculated to quantify the determinism of the model prediction, and its value is normalized. Step 5: Replace the corresponding label word vector with the first-order word prediction vector as input to obtain the second-order word prediction. Quantify the cumulative error between the second-order word prediction and the word label, including: The first-order word prediction vector is used to replace the corresponding label word vector and input into the decoder. The second-order word prediction sequence is obtained through cross-modal interaction. The cross-entropy loss value of the second-order word prediction sequence and the word tag sequence is calculated and quantized based on the cumulative error of generating the second-order prediction from the first-order prediction sequence, and its value is normalized. Step 6: Based on the quantization results, weighted fuse word tags and model predictions to construct pseudo-tags, perform cross-modal interaction again, and output image captions, including: The product factor is calculated based on the model's convergence curve, and the sum of the quantized values ​​is mapped to [0,1] as the word reliability weight. ; Will As the weight of the original word label, As weights for word prediction in the model, the word labels are weighted and fused with the model predictions to construct pseudo-labels; Use pseudo-tags to re-perform cross-modal interaction calculations; The interaction results are linearly transformed, and an activation function is used to map the category with the highest posterior probability to obtain a word prediction sequence, which is then output as an image caption.

2. The robust image caption generation method based on fine-grained noise perception as described in claim 1, characterized in that, In step 2, the grid features and global features of the image are extracted and processed to obtain refined grid features and refined global features, and text features are extracted, including the following steps: The original image is input into the encoder for feature extraction to obtain image grid features and global features; The image mesh features and global features are input into a thinning encoder to extract intramodal correlation information, thereby obtaining thinned mesh features and thinned global features; The original text is input into the encoder for feature extraction to obtain the semantic features of the text encoding.

3. The robust image caption generation method based on fine-grained noise perception as described in claim 2, characterized in that, In step 2, the original image is input into the encoder for feature extraction to obtain image grid features and global features, including: A window self-attention transformer is used as an encoder to extract mesh features; The average pooling of the grid features is calculated as a global feature.

4. The robust image caption generation method based on fine-grained noise perception as described in claim 2, characterized in that, In step 2, the image mesh features and global features are input into the thinning encoder to extract intramodal correlation information, obtaining thinned mesh features and thinned global features, including: The window self-attention mechanism is used to capture the relationship between features within a modality and obtain image features carrying information about intramodal correlation. The image features carrying intramodal correlation information are input into the feedforward neural network and then subjected to layer normalization processing. The final refined grid features and refined global features are obtained from the last layer block.

5. The robust image caption generation method based on fine-grained noise perception as described in claim 1, characterized in that, In step 3, the pre-fused image refines global features and text features, models the correlation between the refined image grid features and the image-text pre-fused features, and realizes cross-modal interaction, including: The image refinement global features are fused into the text features that are currently being input into the decoder; Modal interaction correlation information between image refinement mesh features and image-text pre-fusion features is modeled based on cross-multi-head self-attention.

6. The robust image caption generation method based on fine-grained noise perception as described in claim 5, characterized in that, In step 3, the image refinement global features are fused into the text features to be input to the decoder, including: The semantic features of the text encoding are concatenated with the global features of the image refinement. The concatenated features are then processed using an activation function and subjected to layer normalization. Language masking modeling is performed on the processed spliced ​​features to obtain image-text fusion features carrying interactive information within the text modality.

7. The robust image caption generation method based on fine-grained noise perception as described in claim 5, characterized in that, In step 3, the modal interaction correlation information between the image refinement mesh features based on cross-head self-attention modeling and the image-text pre-fusion features includes: The image-text pre-fusion features are used as query vectors, and the image refinement grid features are used as key and value vectors. These are then input into the decoder for interactive processing to obtain cross-modal interaction information.

Images