Graphic and text feature automatic extraction method based on weak supervised learning
By employing a confidence-driven cross-modal feature interaction and dynamic pseudo-label cyclic optimization strategy, the challenge of aligning image and text features under weak supervision was solved, achieving high-quality image and text feature extraction and semantic alignment. This reduced the cost of manual annotation and improved the robustness and generalization ability of the model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- WUHAN DIGITAL TECH CO LTD
- Filing Date
- 2025-12-25
- Publication Date
- 2026-05-01
AI Technical Summary
In multimodal artificial intelligence systems, existing technologies struggle to achieve stable and accurate alignment of image and text features under weak supervision, especially when there are weak semantic associations and noise mappings between images and text. Traditional methods are unable to reduce the cost of manual annotation and improve the quality of semantic alignment.
By employing a confidence-driven cross-modal feature interaction mechanism and a dynamic pseudo-label cyclic optimization strategy, a confidence matrix is constructed to filter high-confidence association pairs. Combined with contrastive learning training, automatic extraction and semantic alignment of image and text features are achieved.
Achieve high-quality image and text feature extraction and semantic alignment under weak supervision, reduce manual annotation costs, and improve the robustness and generalization ability of the model in multiple styles and scenarios.
Smart Images
Figure CN121962669A_ABST
Abstract
Description
An Automatic Image and Text Feature Extraction Method Based on Weakly Supervised Learning Technical Field
[0001] This invention relates to the fields of artificial intelligence and multimodal information processing technology, and in particular to an automatic method for extracting image and text features based on weakly supervised learning. Background Technology
[0002] In multimodal artificial intelligence systems, the semantic association between images and text is a key capability supporting various tasks such as image-text retrieval, content understanding, image generation, and intelligent moderation. Traditional image-text feature extraction methods typically rely on large-scale, accurate, and one-to-one correspondence of image-text annotation data, achieving alignment between image features and text semantics through end-to-end supervised training.
[0003] However, in practical applications, obtaining massive amounts of accurately labeled data is extremely costly, and in many business scenarios (such as image titles, keyword tags, user-generated text, etc.), images and text only have weak semantic relationships. Even though traditional contrastive learning methods attempt to align image features with semantic features, under weak supervision, they still struggle to reliably achieve accurate alignment between image features and semantic features in the face of numerous noisy mapping relationships between images and text.
[0004] Therefore, there is an urgent need for a technical solution that can automatically and stably extract image and text features and achieve cross-modal alignment under weak supervision and noisy annotation conditions, so as to reduce the cost of manual annotation and improve the semantic alignment quality in weak supervision scenarios, and meet the actual needs of large-scale and diversified image and text data processing. Summary of the Invention
[0005] To address the aforementioned technical problems, this invention proposes an automatic image and text feature extraction method based on weakly supervised learning. This method enables the model to achieve high-quality image and text feature extraction and semantic alignment even in the absence of precise annotations by constructing a confidence-driven cross-modal feature interaction mechanism and a dynamic pseudo-label cyclic optimization strategy.
[0006] An automatic feature extraction method for images and text based on weakly supervised learning is described below:
[0007] S1. Construct a weakly supervised image-text dataset and perform data preprocessing. Collect images and weakly semantically associated texts to construct the dataset. Perform normalization and noise filtering on the images, and perform word segmentation, stop word removal, and word vector initialization preprocessing on the text.
[0008] S2, extract image region-level features and text word-level features. Use an object detection model to extract the image region-level visual feature matrix and use a language model to obtain the text word-level semantic feature matrix. Map the image region features and word-level features to a common shared semantic space to clarify the two types of core features and representations.
[0009] S3, based on the cross-modal attention mechanism, builds a text-image interaction module, inputs two types of feature matrices to calculate semantic association weights and similarities, and generates a confidence matrix of image region-text word matching degree;
[0010] S4. Based on the confidence matrix, select high-confidence association pairs as initial pseudo-labels. Combine contrastive learning training with feature similarity feedback, and iteratively update the confidence matrix and pseudo-labels until the model loss converges.
[0011] S5 outputs shared spatial features of image and text that integrate visual and semantic information, and simultaneously outputs the final matching and alignment results of image regions and text words, providing support for downstream cross-modal tasks.
[0012] As a preferred embodiment of the above technical solution, in step S2,
[0013] The target detection model adopts any one of the following model structures with region-level feature output: Faster R-CNN, DETR, and YOLO, to finely express the local semantics of the image;
[0014] The language model adopts any one of the Transformer architectures of Qwen, BERT, and RoBER to obtain word-level token semantic representations;
[0015] The method of mapping to a common shared semantic space is to project image region features and text word features onto a unified d-dimensional vector space through a multilayer perceptron, making the two types of features comparable.
[0016] As a preferred embodiment of the above technical solution, in step S3, the confidence matrix is constructed by fusing three types of core information to accurately represent the semantic matching confidence of image regions and text words, specifically including:
[0017] Cross-modal similarity in shared space: a quantification of the semantic association between image region features and text word features in the image-text shared space;
[0018] Cross-modal attention weights: calculated using a two-stream attention mechanism or a cross-modal attention mechanism, reflecting the key attention weights when two types of modal features interact;
[0019] Cross-modal consistency information: A quantitative metric for verifying the consistency between image regions and the semantic expression of text words, used to reduce noise interference.
[0020] As a preferred embodiment of the above technical solution, in step S3, the fusion formula for the confidence matrix C(i,j) is calculated as follows:
[0021] ;
[0022] Where C(i,j) represents the element in the i-th row and j-th column of the confidence matrix, with a value range of [0,1]. The larger the value, the higher the semantic matching degree between the corresponding region and the word; σ is the activation function, used to map the fusion result to the [0,1] interval; α, β, and γ are all fusion coefficients, used to adjust the contribution weight of the three types of core information in the confidence calculation, with a value range of 0.1 to 2, which can be adaptively adjusted according to the characteristics of the actual dataset; S is the cross-modal similarity quantification value of the shared space; A_img is the attention weight corresponding to the image modality; A_txt is the attention weight corresponding to the text modality.
[0023] As a preferred embodiment of the above technical solution, in step S4, the pseudo-labels include region-level pseudo-labels, word-level pseudo-labels, image-text pair-level pseudo-labels, and alignment masks, all of which are stored in matrix or probability form and input into the model for supervised learning.
[0024] The region-level pseudo-labels are used to represent the text words most likely corresponding to each image region. The pseudo-label matrix is Y_region ∈ R^(N × L), where N is the number of given image regions and L is the number of given text words.
[0025] The word-level pseudo-labels are used to represent the regions of the image corresponding to each text word. The word-level pseudo-label matrix is Y_token ∈ R^(L × N), where N is the number of image regions of a single text and L is the number of words of a single text.
[0026] The image-text pair pseudo-label is used to represent the overall matching rate of the image-text pair. The overall matching degree of the image-text pair is generated based on the statistical characteristics of the confidence matrix, and the output is a scalar or a binary probability.
[0027] The alignment mask is used to mark regions that are "low confidence but need to be ignored", and an alignment mask matrix is output.
[0028] As a preferred embodiment of the above technical solution, contrastive learning based on pseudo-labels is implemented using a weighted InfoNCE loss function. This approach enhances cross-modal feature alignment through pseudo-label association. The weights of the loss function are defined as follows:
[0029] w(i,j)=C(i,j)γ
[0030] Where w(i,j) represents the contrastive learning loss weight corresponding to the i-th image region and the j-th text word, used to adjust the contribution of this sample to the model training; C(i,j) represents the element in the i-th row and j-th column of the confidence matrix, that is, the semantic matching confidence of the corresponding image region and text word; γ is the weight adjustment coefficient, used to control the degree of weight amplification of high confidence samples, and the value range is 1 to 4.
[0031] The beneficial effects of this invention are as follows:
[0032] 1. Achieving high-quality image-text alignment under weak supervision: Improve alignment stability by using confidence matrix and dynamic pseudo-labels, enabling the model to reliably learn cross-modal semantic mapping even when precise annotations are lacking.
[0033] 2. Improve model generalization ability: Dynamic pseudo-label loop optimization reduces noise accumulation, enabling the model to maintain robust alignment performance in multiple styles, scenarios, and domains.
[0034] 3. Reduce manual annotation costs: Large-scale automatic training can be carried out using weak or no annotations, without relying on precise manual annotation data. Attached Figure Description
[0035] Figure 1 is a schematic diagram of the process of the present invention.
[0036] Figure 2 is a schematic diagram of cross-modal feature interaction and confidence matrix generation.
[0037] Figure 3 is a schematic diagram of the dynamic pseudo-label generation and iterative optimization mechanism. Detailed Implementation
[0038] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0039] Figure 1 illustrates an automatic image and text feature extraction method based on weakly supervised learning, which includes the following steps:
[0040] Step 1: Construct a weakly supervised graph-text dataset and perform data preprocessing.
[0041] Weakly supervised image-text datasets are constructed by downloading open-source image-text datasets or collecting images and corresponding weak semantic text such as titles, keywords, and short text descriptions. Then, image processing is performed on the images, including size normalization, color standardization, and random cropping. Text processing is performed on the text, including word segmentation, noise reduction, spell correction, and stop word filtering. To improve robustness, images are enhanced using random occlusion and brightness perturbation, while text is enhanced using weak perturbation strategies such as synonym replacement and random deletion.
[0042] Step 2: Extract image region-level features and text word-level features
[0043] 1. Image region feature extraction: A visual encoder with region-level output capability, such as Faster R-CNN, YOLO, DETR, or other object detection models, is used to decompose the image into multiple regions or patches and extract region vectors with dimensions of 256–2048. The output results are as follows:
[0044] ;
[0045] 2. Text word-level feature extraction: Input the segmented text into a Transformer-based language model (such as Qwen, BERT, RoBERTa, etc.) to extract word-level token representations.
[0046] ;
[0047] 3. Project the features into the shared space. Input the region features and word-level features into the projection layer (two MLP layers + activation function) to obtain a d-dimensional shared space vector: d ∈ [256, 768], temperature coefficient τ ∈ [0.05, 0.3].
[0048] Step 3: Build a text-image interaction module based on a cross-modal attention mechanism. Input two types of feature matrices to calculate semantic association weights and similarities, and generate a confidence matrix for image region-text word matching, as shown in Figure 2.
[0049] 1. Calculate the cross-modal similarity matrix. The similarity between the region vector r and the word vector t can be expressed as: s(r,t)=cos(r,t) / τ, and the similarity matrix is obtained as: S ∈ R^(N_region × N_token);
[0050] 2. Use a two-stream attention network model (such as DANet) to calculate the text weight (A_image) and the region weight (A_text) of the text attention.
[0051] 3. Construct a confidence matrix by fusing three types of core information to accurately represent the semantic matching confidence of image regions and text words. The element C(i,j) in the confidence matrix (corresponding to the matching confidence of the i-th image region and the j-th text word) is calculated using the following fusion formula:
[0052] ;
[0053] C(i,j): The element in the i-th row and j-th column of the confidence matrix, with a value range of [0,1]. The larger the value, the higher the semantic matching degree between the corresponding region and the word.
[0054] σ: Activation function (such as the Sigmoid function), used to map the fusion result to the [0,1] interval to ensure the reasonableness of the confidence level;
[0055] α, β, γ: Fusion coefficients, used to adjust the contribution weight of the three types of core information in the confidence calculation. The typical value range is 0.1 to 2, and it can be adaptively adjusted according to the characteristics of the actual dataset.
[0056] S: Cross-modal similarity metric of the shared space;
[0057] A_img: Attention weights corresponding to the image modality;
[0058] A_txt: Attention weights corresponding to the text modality.
[0059] Step 4: Based on the confidence matrix, select high-confidence association pairs as initial pseudo-labels. Combine contrastive learning training with feature similarity feedback to iteratively update the matrix and pseudo-labels until the model loss converges, as shown in Figure 3.
[0060] 1. Pseudo-tag generation methods: Pseudo-tags include region-level pseudo-tags, word-level pseudo-tags, image-text pair-level pseudo-tags, and alignment masks, specifically as follows:
[0061] 1) Region-level pseudo-labels: For each region ri, select the word tj with the highest confidence as the soft label: y_region(i)=softmax(C[i,*] / T);
[0062] 2) Word-level pseudo-tags: Similar to region-level tags, the most relevant regions are selected in reverse;
[0063] 3) Image-text alignment pseudo-tags: the probability of positive alignment between the entire image and the entire text;
[0064] 4) Alignment mask: Used to mark regions with "low confidence but need to be ignored" to reduce noise interference and output an alignment mask matrix.
[0065] 2. Dynamic threshold mechanism: To avoid noise accumulation, this invention adopts a threshold increment strategy:
[0066] ;
[0067] Where k is the number of training rounds, θ ∈ [0.6, 0.9], pseudo-labels below the threshold are discarded, and the medium confidence level is incorporated into the loss using soft labels.
[0068] 3. Periodic pseudo-label recalculation: The confidence matrix and pseudo-labels are recalculated every T_epoch (1–5 epochs) to gradually reduce weak supervision noise.
[0069] 4. Based on pseudo-label contrastive learning, pseudo-labels are used as weighted inputs into the InfoNCE contrastive loss:
[0070] L = - w(i,j) · log( exp(s(i,j)) / Σ_k exp(s(i,k)) )
[0071] The weights are:
[0072] w(i,j)=C(i,j)^γ, γ ∈ [1,4].
[0073] Step 5: Output the image-text shared spatial features that integrate visual and semantic information, and simultaneously output the final matching and alignment results of image regions and text words to provide support for downstream cross-modal tasks.
[0074] After completing the model training and pseudo-label optimization steps outlined above, the inference phase begins. This step aims to output image-text shared spatial features that integrate visual and semantic information based on the input image and text, and simultaneously provide the final matching and alignment results between image regions and text words to support downstream cross-modal retrieval, matching, and understanding tasks.
[0075] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for automatic extraction of image and text features based on weakly supervised learning, characterized in that: The specific process is as follows: S1, construct a weakly supervised image-text dataset and perform data preprocessing. Collect images and weakly semantically associated text to construct the dataset. Perform normalization and noise filtering on the images, and perform word segmentation, stop word removal, and word vector initialization preprocessing on the text; S2, extract image region-level features and text word-level features. Use an object detection model to extract the image region-level visual feature matrix, and use a language model to obtain the text word-level semantic feature matrix. Map the image region features and word-level features to a common shared semantic space to clarify the two types of core features and representations; S3: Based on a cross-modal attention mechanism, a text-image interaction module is built. Two types of feature matrices are input to calculate semantic association weights and similarities, generating a confidence matrix for image region-text word matching. S4: Based on the confidence matrix, high-confidence association pairs are selected as initial pseudo-labels. Combining contrastive learning training and feature similarity feedback, the confidence matrix and pseudo-labels are iteratively updated until the model loss converges. S5: The shared spatial features of text and images, which integrate visual and semantic information, are output, and the final matching and alignment results of image regions and text words are output simultaneously, providing support for downstream cross-modal tasks.
2. The method for automatic extraction of image and text features based on weakly supervised learning according to claim 1, characterized in that: In step S2, the object detection model adopts any one of the following model structures with region-level feature output: Faster R-CNN, DETR, and YOLO, to finely express the local semantics of the image; the language model adopts any one of the following Transformer architectures: Qwen, BERT, and RoBER, to obtain word-level token semantic representations; the mapping to a common shared semantic space is achieved by projecting image region features and text word features onto a unified d-dimensional vector space through a multilayer perceptron, making the two types of features comparable.
3. The method for automatic extraction of image and text features based on weakly supervised learning according to claim 1, characterized in that: In step S3, the confidence matrix is constructed by fusing three types of core information to accurately represent the semantic matching confidence of image regions and text words. Specifically, it includes: cross-modal similarity of the shared space: a quantitative value of the semantic association between image region features and text word features in the image-text shared space; weight of cross-modal attention: calculated by a dual-stream attention mechanism or a cross-modal attention mechanism, reflecting the key attention weight when the two types of modal features interact; and cross-modal consistency information: a quantitative indicator to verify the semantic consistency between image regions and text words, used to reduce noise interference.
4. The method for automatic extraction of image and text features based on weakly supervised learning as described in claim 1, characterized in that: In step S3, the fusion formula for the confidence matrix C(i,j) is calculated as follows: Where C(i,j) represents the element in the i-th row and j-th column of the confidence matrix, with a value range of [0,1]. The larger the value, the higher the semantic matching degree between the corresponding region and the word. σ is the activation function, used to map the fusion result to the [0,1] interval. α, β, and γ are all fusion coefficients, used to adjust the contribution weight of the three types of core information in the confidence calculation, with a value range of 0.1 to 2, which can be adaptively adjusted according to the characteristics of the actual dataset. S is the cross-modal similarity metric of the shared space. A_img is the attention weight corresponding to the image modality. A_txt is the attention weight corresponding to the text modality.
5. The method for automatic extraction of image and text features based on weakly supervised learning according to claim 1, characterized in that: In step S4, the pseudo-labels include region-level pseudo-labels, word-level pseudo-labels, image-text pair-level pseudo-labels, and alignment masks, all stored in matrix or probabilistic form and input into the model for supervised learning. The region-level pseudo-labels represent the most likely text words corresponding to each image region, with a pseudo-label matrix of Y_region ∈ R^(N × L), where N is the number of given image regions and L is the number of given text words. The word-level pseudo-labels represent the regions of the image corresponding to each text word, with a word-level pseudo-label matrix of Y_token ∈ R^(L × N), where N is the number of image regions for a single text and L is the number of words for a single text. The image-text pair-level pseudo-labels represent the overall matching rate of the image-text pair, generating the overall matching degree based on the statistical characteristics of the confidence matrix, and outputting a scalar or binary probability. The alignment mask is used to mark regions with "low confidence but need to be ignored," outputting an alignment mask matrix.
6. The method for automatic extraction of image and text features based on weakly supervised learning according to claim 5, characterized in that: Based on pseudo-labels, contrastive learning is implemented using a weighted InfoNCE loss function. Pseudo-label association enhances cross-modal feature alignment. The weight term of the loss function is defined as: w(i,j) = C(i,j)γ, where w(i,j) represents the contrastive learning loss weight corresponding to the i-th image region and the j-th text word, used to adjust the contribution of this sample to model training; C(i,j) represents the element in the i-th row and j-th column of the confidence matrix, i.e., the semantic matching confidence of the corresponding image region and text word; γ is a weight adjustment coefficient used to control the degree of weight amplification for high-confidence samples, with a value ranging from 1 to 4.