Image-text content review method based on multi-modal large model

Abstract

The application discloses a method for image-text content review based on a multi-modal large model, relates to the technical field of image-text content review, and significantly improves the accuracy and efficiency of image-text content review by constructing a pyramid label system and implementing a phased dynamic adjustment strategy. In the early stage of training, coarse-grained labels are used to quickly establish the basic classification ability of the model, avoiding the low training efficiency and overfitting problem caused by the early introduction of fine-grained labels in traditional methods. When the medium-grained labels are introduced in the middle stage, the model can further refine the violation type identification, especially improving the identification accuracy of easily confused violation categories. When fine-grained labels are used in the later stage, the identification ability of the model for specific violation elements is significantly enhanced, and the model can accurately locate subtle violation content. This coarse-to-fine label system adjustment mechanism not only improves the review performance of the model, but also shortens the model training period and reduces the consumption of computing resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image-text content review, in particular to an image-text content review method based on a multi-modal large model. BACKGROUND

[0002] With the development of the Internet, the spread speed and scale of image-text content are growing exponentially. However, a large amount of image-text content may contain illegal information, which poses a serious threat to the physical and mental health of users, social order and platform ecology. Therefore, efficient and accurate image-text content review technology has become an urgent need for Internet platforms and regulatory agencies. Traditional manual review methods are inefficient, costly, and difficult to cope with massive data. Automatic review methods based on rules or single modal have the problems of poor generalization ability and high misjudgment rate. Currently, multi-modal large models have shown significant advantages in image-text content understanding tasks by fusing image and text features.

[0003] Existing image-text content review technology usually directly uses pre-trained models to generate pseudo-labels when facing unlabeled data, but lacks effective pseudo-label quality evaluation mechanisms, resulting in low-quality pseudo-labels mixed into training data, affecting model performance. In terms of label system construction, most methods use single-granularity labels, which cannot dynamically adjust the refinement degree of labels according to the model training stage, making it difficult for the model to quickly establish basic classification ability in the early training stage, and unable to further improve the review accuracy in the later training stage. In addition, existing multi-modal training methods fail to fully utilize the advantages of different granularity labels, making it difficult to effectively fuse the complex relationships between image and text features, resulting in insufficient model recognition ability for comprehensive illegal content of image-text.

[0004] In summary, the existing image-text content review technology has obvious defects in pseudo-label generation and screening, label system construction, and multi-modal feature fusion training, which cannot meet the growing demand for network content review. Therefore, a kind of image-text content review method is needed, which can efficiently process unlabeled data, dynamically adjust label granularity and fully fuse multi-modal features, to improve the review accuracy and efficiency, and adapt to the changing network environment. SUMMARY

[0005] The purpose of the present application is to make up for the shortcomings of the prior art, and provide a multi-modal large model-based image-text content review method, which can significantly improve the accuracy and efficiency of image-text content review by constructing a pyramid label system and implementing a phased dynamic adjustment strategy. In the early stage of training, coarse-grained labels are used to quickly establish the basic classification ability of the model, avoiding the low training efficiency and overfitting problem caused by the early introduction of fine-grained labels in traditional methods. When medium-grained labels are introduced in the middle stage, the model can further refine the violation type identification, especially improving the identification accuracy of easily confused violation categories. When fine-grained labels are used in the later stage, the model's identification ability of specific violation elements is significantly enhanced, and it can accurately locate subtle violation content. This coarse-to-fine label system adjustment mechanism not only improves the review performance of the model, but also shortens the model training cycle and reduces the consumption of computing resources.

[0006] To solve the above technical problems, the present application provides the following technical scheme: a multi-modal large model-based image-text content review method, the specific steps of which are as follows:

[0007] S100, multi-modal pseudo-label generation: using a multi-modal large model to generate pseudo-labels for unlabeled image-text data, fusing image features and text features to generate an initial pseudo-label set;

[0008] S200, pseudo-label quality evaluation and screening: designing a confidence evaluation index to score the quality of the pseudo-labels, screening high-confidence pseudo-labels, and constructing a high-quality pseudo-label dataset for subsequent training;

[0009] S300, label system construction: dividing the high-quality pseudo-label dataset of S200 into three levels and dynamically adjusting the label granularity in stages;

[0010] S400, multi-granularity contrastive learning training: using the high-quality pseudo-label dataset of S200 and the label system adjustment information, combined with a contrastive learning strategy, to perform semi-supervised training on the multi-modal large model;

[0011] S500, model iteration optimization: inputting the image-text content to be reviewed into the trained multi-modal large model, and the model outputs the review result according to the learned knowledge and features, completing the image-text content review.

[0012] Further, the S100 specifically includes:

[0013] The image in the unlabeled image-text data is feature-extracted by a pre-trained visual encoder to extract the global features and local features of the image, forming a visual feature vector of the image;

[0014] The text in the unlabeled text data is feature-extracted by a pre-trained text encoder to extract the syntactic features of the text, forming a semantic feature vector of the text;

[0015] The visual feature vector of the image and the semantic feature vector of the text are weighted and fused through the attention mechanism, and based on the weighted and fused multi-modal features, an initial pseudo label set is generated through a prediction layer of a multi-modal large model, each pseudo label in the initial pseudo label set including an image violation category, a text violation category and a comprehensive violation judgment result of the image and the text.

[0016] Further, the S200 designs a confidence evaluation index to score the quality of the pseudo label: C(L p ) = a x P p (L p ) + β x Sim f (L p ) + γ x Sim c (L p ), wherein C(L p ) represents the confidence score of the pseudo label L p , and the value range is [0, 1], and the higher the score, the higher the credibility of the pseudo label, P p (L p ) is a probability value of the pseudo label L p predicted by the multi-modal large model, and the value range is [0, 1], which is output by the prediction layer of the multi-modal large model, and reflects the determination degree of the model on the pseudo label prediction, Sim f (L p ) is the similarity of the pseudo label L p corresponding to the image-text feature and the labeled similar image-text feature, which is calculated by the cosine similarity formula, and the value range is [0, 1], and the formula is: F p is the image-text fusion feature vector corresponding to the pseudo label L p , F s is the fusion feature vector of the labeled similar image-text, ||·|| represents the length of the vector, and Sim c (L p ) is the cross-modal consistency score of the image violation category and the text violation category in the pseudo label L p , and the value range is [0, 1], which is obtained by calculating the cosine similarity of the image and text violation category feature vectors, a, β, γ are weight coefficients, satisfying a + β + γ = 1, and a, β, γ ≥ 0, for adjusting the importance of different evaluation dimensions, the pseudo label with a confidence score C(L p )>0.8 is screened out to construct a high-quality pseudo label dataset.

[0017] Further, the S300 has three levels: coarse granularity, medium granularity and fine granularity, wherein:

[0018] Coarse-grained label generation: For high-quality pseudo-label data, use a multi-modal large model to predict violations of large categories as coarse-grained high-confidence samples, input to a multi-modal large model to establish a model foundation, and calculate the accuracy of the model;

[0019] Medium-grained label generation: For coarse-grained high-confidence samples, refine the predicted violations, and use medium-grained high-confidence samples to refine the violation types when the model accuracy reaches the first preset value in the contrast learning training;

[0020] Fine-grained label generation: For medium-grained high-confidence samples, locate specific violation elements, and use fine-grained labels to improve audit accuracy when the accuracy reaches the second preset value and tends to be stable.

[0021] Further, the S300 sets the current training round as n, the accuracy of the model after the nth training as A(n), defines the first preset accuracy threshold as A1 and the second preset accuracy threshold as A2, and 0 < A1 < A2 < 1; At the same time, an accuracy fluctuation coefficient k is set to measure the stability of the accuracy, and the smaller the k, the more stable the accuracy. The And when n = 1, k(1) = 0, where A(n) represents the accuracy of the model after the nth training, with a value range of [0, 1], and the higher the value, the higher the accuracy of the model in judging the content of the text after the training, A(n-1) represents the accuracy of the model after the (n-1)th training, TP(n) represents the number of text samples that the model correctly predicts as violations after the nth training, i.e. the number of samples that are actually violations and are also judged as violations by the model, TN(n) represents the number of text samples that the model correctly predicts as compliant after the nth training, i.e. the number of samples that are actually compliant and are also judged as compliant by the model, FP(n) represents the number of text samples that the model incorrectly predicts as violations after the nth training, i.e. the number of samples that are actually compliant but are misjudged as violations by the model, FN(n) represents the number of text samples that the model incorrectly predicts as compliant after the nth training, i.e. the number of samples that are actually violations but are misjudged as compliant by the model, k(n) is the fluctuation coefficient of the accuracy in the nth training, the label granularity level is represented by Level, and n e {1, 2, 3}, where 1 represents coarse-grained, 2 represents medium-grained, and 3 represents fine-grained.

[0022] Further, the S300 dynamically adjusts the label granularity in stages, which is specifically:

[0023] When n ≤ N1 and A(n) < A1, Level(n) = 1, at this time, coarse-grained labels are used to establish the basic classification ability of the model, and N1 is the set initial number of training rounds;

[0024] When A1≤A(n)<A2 and k(n)≤K1: Level(n)=2, introduce medium-grained labels, refine violation type identification, K1 is the stable threshold of medium-grained switching and K1=0.05, and when k(n)>K1, maintain Level(n)=1;

[0025] When A(n)≥A2 and k(n)≤K2: Level(n)=3, use fine-grained labels, improve audit accuracy, and when K2<k(n)<K1, maintain Level(n)=2 to ensure that the model improves performance in a stable state, K2 is the stable threshold of fine-grained switching and K2=0.03;

[0026] After each completion of label granularity level adjustment, the current Level(n) is fed back to S400 as label system adjustment information.

[0027] Further, the semi-supervised training step of S400 is:

[0028] Obtain a labeled sample set S l Each sample in the set has a manually labeled label, which is used to provide supervision information for the model, and a high-quality pseudo-label data set S p is used as complementary data to the labeled samples, receive label system adjustment information from S300, obtain the current label granularity level Level(n), and for any sample x l in the labeled sample set S i , obtain the visual feature vector and the semantic feature vector of the image and text, respectively, and obtain the comprehensive feature vector after feature fusion, and the corresponding accurate label is y i For any sample x p in the high-quality pseudo-label data set S j , also obtain the comprehensive feature vector after fusion, and the corresponding pseudo-label is y j ;

[0029] Semi-supervised training of the multi-modal large model using a contrastive learning loss function L c The loss function where N represents the number of samples contained in each training batch, k represents the kth sample in the current batch, sim(·,·) is a cosine similarity calculation function for measuring the similarity between two feature vectors, f k is the feature vector of the kth sample, represents a positive sample feature vector of the same class as the sample x k , and when constructing positive samples, positive samples are simultaneously selected from S l and Sp In search of x k Label y k The same sample, get its feature vector as Represent x k Different class of negative sample feature vector, from S l And S p Filter out the sample whose label is different from y k Get its feature vector as negative sample feature vector M represents the number of negative samples, m represents the serial number of negative samples, ω(y k ,Level) is the weight coefficient related to the sample label y k And the current label granularity level Level, τ is a hyperparameter and the initial value is 0.1, and is dynamically adjusted according to the label granularity level Level: when Level(n)=1, τ=0.1, when Level(n)=2, τ=0.05, when Level(n)=3, τ=0.03.

[0030] Further, the S300 gives different weights to the samples according to the current label granularity level Level(n) in the multi-modal large model training process:

[0031] When Level(n)=1, the same weight is given to all samples, that is, the contribution degree of each sample in calculating the loss function is the same, that is, ω(y k ,Level)=1;

[0032] When Level(n)=2, y k Belongs to the medium-grained label category, then ω(y k ,Level)=1.5, otherwise 1;

[0033] When Level=3, y k Belongs to the fine-grained label category, then ω(y k ,Level)=2.0, otherwise 1.

[0034] Compared with the prior art, the image-text content review method based on the multi-modal large model has the following beneficial effects:

[0035] One, the pyramid label system is constructed and the phased dynamic adjustment strategy is implemented, the precision and efficiency of image-text content review are significantly improved, in the early training stage, the coarse-grained label is used to quickly establish the basic classification ability of the model, the low training efficiency and overfitting problem caused by the early introduction of fine-grained labels in traditional methods are avoided, when the medium-grained label is introduced in the middle period, the model can further refine the violation type identification, especially improve the identification accuracy of the easily confused violation categories, and when the fine-grained label is used in the later period, the identification ability of the model to specific violation elements is significantly enhanced, and the fine violation content can be accurately positioned. This label system adjustment mechanism from coarse to fine not only improves the auditing performance of the model, but also shortens the model training cycle and reduces the consumption of computing resources.

[0036] Two, the multi-granularity contrast learning training method proposed in the present application effectively solves the deficiencies of the prior art in multi-modal feature fusion, greatly improves the identification ability of the model to image-text comprehensive violation content, and through the contrast learning strategy, the model can learn the deep correlation information between image and text features, so that the image-text content is more accurately represented in the feature space. When processing unlabeled data, the method of the present application fully utilizes the value of a large amount of unlabeled data through high-quality pseudo-label screening and dynamic weight adjustment, and the model performance can still be stably improved under the condition of limited labeled data. According to the actual demand, the label granularity and training strategy can be flexibly adjusted, and the method is suitable for image-text content review in different fields and scenes.

[0037] Other advantages, objects, and features of the present application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following or can be learned from practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0039] Figure 1 The operation flowchart of the image-text content review method based on the multi-modal large model;

[0040] Figure 2 The step framework diagram of the image-text content review method based on the multi-modal large model. DETAILED DESCRIPTION

[0041] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0042] Example 1

[0043] This embodiment provides a method for reviewing text and image content based on a multimodal large model, such as... Figure 1 As shown, this method achieves efficient and accurate review of image and text content through steps such as multimodal pseudo-label generation, pseudo-label quality assessment and screening, label system construction, multi-granularity comparative learning training, and model iterative optimization. Its core lies in using a multimodal large model to fuse image and text features to generate pseudo-labels, screening high-quality pseudo-labels through confidence evaluation indicators, constructing a three-level label system that is dynamically adjusted in stages, and combining it with a comparative learning strategy for semi-supervised training, thereby significantly improving the model's accuracy and generalization ability in identifying illegal image and text content, and making it suitable for complex image and text content review scenarios on Internet platforms.

[0044] First, the multimodal pseudo-label generation stage (S100) is entered. In this stage, pre-trained visual and text encoders are used to extract features from the unlabeled image and text data, respectively. These features are then fused using an attention mechanism, and the multimodal large model generates pseudo-labels. For images in the unlabeled image and text data, the pre-trained visual encoder extracts global and local features. Global features reflect the overall semantic information of the image, and these two types of features together form the image's visual feature vector, comprehensively describing the image's visual information. For text data, the pre-trained text encoder extracts syntactic features, which reflect the text's semantic content. The model constructs a semantic feature vector of the text by analyzing the structure and semantic relationships of the images, which represents the semantic content of the text. An attention mechanism is used to weightedly fuse the visual feature vector of the image and the semantic feature vector of the text. The attention mechanism can automatically learn the degree of correlation between image and text features and assign different weights to different feature components, so that the model pays more attention to the features that are important for violation judgment. Based on the weighted fused multimodal features, an initial set of pseudo-labels is generated through the prediction layer of the multimodal large model. Each pseudo-label contains the image violation category, the text violation category, and the comprehensive image and text violation judgment result, thereby realizing multi-dimensional prediction of the violation of image and text content.

[0045] Then, the pseudo-label quality assessment and screening stage (S200) begins. This stage designs a confidence assessment index to score the quality of the pseudo-labels, comprehensively considering multiple dimensions such as model prediction probability, image-text feature similarity, and cross-modal consistency to fully evaluate the credibility of the pseudo-labels. The pseudo-label quality scoring formula is C(L... p )=α×P p (Lp )+ β x Sim f (L p )+ γ x Sim c (L p ), wherein C(L p ) represents the confidence score of the pseudo label L p , with a value range of [0, 1], and the higher the score, the more reliable the pseudo label, P p (L p ) is the probability value of the multi-modal large model predicting the pseudo label L p , which is output by the prediction layer of the model, and reflects the determination degree of the model on the prediction, and the higher the probability value, the more accurate the prediction result of the model, Sim f (L p ) is the similarity of the pseudo label L p to the similar image-text feature, which is calculated by the cosine similarity formula , wherein F p is the image-text fusion feature vector corresponding to the pseudo label L p , F s is the fusion feature vector of the labeled similar image-text, and ||·|| represents the length of the vector. The similarity reflects the similarity degree of the image-text feature corresponding to the pseudo label to the known labeled data feature, and the higher the similarity degree, the more reliable the pseudo label may be. Sim c (L p ) is the cross-modal consistency score of the image violation category and the text violation category in the pseudo label L p , which is obtained by calculating the cosine similarity of the image and text violation category feature vectors, and is used to measure the consistency between the image and text violation judgment. The higher the consistency, the more reasonable the pseudo label. α, β, γ are weight coefficients, satisfying α+β+γ=1 and α, β, γ≥0, for adjusting the importance of different evaluation dimensions, which can be set according to actual needs. The confidence score of each pseudo label is calculated by the confidence evaluation index, and the pseudo labels with a confidence score greater than a set threshold are selected to construct a high-quality pseudo label dataset, providing reliable samples for subsequent training.

[0046] Then, enter the label system construction phase (S300), divide the high-quality pseudo-label data set into three levels of coarse granularity, medium granularity and fine granularity, and dynamically adjust the label granularity in stages. The principle is to gradually refine the label granularity according to the accuracy and stability in the model training process, so that the model can adapt to different learning difficulties at different training stages, thereby improving the training effect. Among them, when generating coarse-grained labels, high-quality pseudo-label data is used to predict the violation category by a multi-modal large model, and the high-confidence samples of coarse-grained labels are input into the multi-modal large model to establish the model foundation and calculate the model accuracy. In the early stage of training, the model has weak recognition ability for violation content, and the coarse-grained violation category can help the model quickly master the basic classification framework and avoid the difficulty of model learning due to too fine labels. When generating medium-grained labels, the high-confidence samples of coarse-grained labels are used to refine the predicted violation behavior, and the high-confidence samples of medium-grained labels are used. When the accuracy reaches the first preset value in the contrast learning training, introduce the medium-grained label to refine the violation type. At this time, the model has a certain basic classification ability, and the medium-grained label can help the model further distinguish more specific violation behaviors and improve the recognition accuracy of the violation type. When generating fine-grained labels, the high-confidence samples of medium-grained labels are used to locate specific violation elements. When the accuracy reaches the second preset value and tends to be stable, the fine-grained label is used to improve the review accuracy. At this time, the performance of the model is relatively stable, and the fine-grained label can accurately locate the specific violation elements in the image-text content to achieve more detailed review. In order to more accurately judge when to adjust the label granularity, set the current training round as n, the accuracy of the model after the nth training as A(n), define the first preset accuracy threshold as A1, the second preset accuracy threshold as A2 and 0 TP(n) represents the number of image-text samples correctly predicted as violation by the model after the rth training, i.e. the number of samples that are actually violation and are also judged as violation by the model, TN(n) represents the number of image-text samples correctly predicted as compliance by the model after the nth training, i.e. the number of samples that are actually compliant and are also judged as compliant by the model, FP(n) represents the number of image-text samples incorrectly predicted as violation by the model after the nth training, i.e. the number of samples that are actually compliant but are misjudged as violation by the model, and FN(n) represents the number of image-text samples incorrectly predicted as compliance by the model after the nth training, i.e. the number of samples that are actually violation but are misjudged as compliance by the model. This formula comprehensively considers the correct prediction and incorrect prediction of the model, and accurately reflects the review judgment accuracy of the model after this round of training. The accuracy fluctuation coefficient k is When n = 1, k (1) = 0 is used to measure the change range of accuracy between adjacent training rounds, so as to judge the stability of model training; the specific rules of the phased dynamic adjustment of the label granularity are as follows: when n≤N1 and A (n) <A1, Level (n) = 1, the coarse-grained label is used to establish the basic classification ability of the model, wherein N1 is the initial training round number set in the early training of the model, the model needs to master the basic classification framework first, and the coarse-grained label can meet this demand, avoiding the model from falling into a complex learning task due to too fine labels, resulting in low training efficiency, when A1≤A (n) <A2 and k (n) ≤K1, the medium-grained label is introduced to refine the violation type identification, wherein K1 is the stability threshold of medium-grained switching, at this time the accuracy of the model has reached a certain level, and the training tends to be stable, the introduction of the medium-grained label can enable the model to further learn more specific violation types and improve the identification ability, if k (n) >K1, Level (n) = 1 is maintained, because the accuracy fluctuates greatly at this time, the model training is not stable enough, and the continued use of the coarse-grained label can enable the model to stabilize first, avoiding the performance fluctuation of the model caused by the introduction of more fine-grained labels, when A (n) ≥A2 and k (n) ≤K2, the fine-grained label is used to improve the auditing accuracy, wherein K2 is the stability threshold of fine-grained switching and K2 <K1, at this time the accuracy of the model is already high, and the training is very stable, the fine-grained label can enable the model to accurately locate the violation elements and realize more precise auditing, if K2 <k (n) <K1, Level (n) = 2 is maintained, to ensure that the model improves the performance in a stable state and avoid the reckless promotion of the label granularity, which affects the model training effect, after adjusting the label granularity level each time, the current Level (n) is fed back to the subsequent multi-grained contrastive learning training stage (S400) as the label system adjustment information, so that the model is trained according to the current label granularity in the stage.

[0047] After that, enter the multi-grained contrastive learning training stage (S400), and use the labeled sample set and the high-quality pseudo-label data set to combine the contrastive learning strategy and the label system adjustment information to perform semi-supervised training on the multi-modal large model, so as to improve the fusion ability of the model to the text and image features and the identification ability of the model to the violation content, specifically: obtaining the labeled sample set S l Each sample in the set has a manually labeled label, which is used to provide supervision information for the model to ensure that the learning direction of the model is correct, and the high-quality pseudo-label data set S p As complementary data of the labeled sample, the value of a large amount of unlabeled data is fully utilized, and the training sample amount of the model is expanded, the label system adjustment information is received from the label system construction stage to obtain the current label granularity level Level (n), and any sample x in the labeled sample set S l ​i The visual feature vectors of their images are obtained respectively. semantic feature vectors of text The comprehensive feature vector is obtained after feature fusion. The corresponding accurate label is y i For high-quality pseudo-label dataset S p Any sample x in j Similarly, obtain the fused comprehensive feature vector. The corresponding pseudo-label is y j The contrastive learning loss function L is used. c Semi-supervised training is performed on a large multimodal model, and the specific formula for the loss function is as follows: Where N represents the number of samples in each training batch, k represents the k-th sample in the current batch, sim(·,·) is the cosine similarity calculation function, used to measure the similarity between two feature vectors. It determines the directional consistency by calculating the cosine value of the feature vectors, thus reflecting the similarity of the features. k It is the feature vector of the k-th sample. Representative and sample x k The feature vectors of positive samples of the same category are simultaneously derived from S when constructing positive samples. l and S p Searching for x k Tag y k For the same sample, obtain its feature vector as This allows the model to learn the common features of samples of the same type. Representative and sample x k Feature vectors of negative samples from different categories, from S l and S p Filter out tags and y k For different samples, their feature vectors are obtained as negative sample feature vectors. By comparing with negative samples, the model distinguishes the feature differences between different categories. M represents the number of negative samples, ω(y k Level(n) is related to the sample label y kWeight coefficient related to the current label granularity level Level, used to adjust the contribution degree of samples in the loss function according to different label granularity and label categories, τ is a hyperparameter and the initial value is 0.1, and is dynamically adjusted according to the label granularity level Level: when Level(n) = 1, τ = 0.1, when Level(n) = 2, τ = 0.05, when Level(n) = 3, τ = 0.03, in the training process of the multi-modal large model, different weights are given to the samples according to the current label granularity level Level(n): when Level(n) = 1, the model is in the early stage of training, and the main task is to establish basic classification ability, so the same weight ω(y k , Level) = 1 is given to all samples, and the contribution degree of each sample in the calculation of the loss function is the same, which ensures that the model can learn the basic features of all types of samples, when Level(n) = 2, the model has certain basic classification ability, and needs to further refine the identification of violation types, so the sample y k belonging to the medium-grained label category is given a weight ω(y k , Level) = 1.5, and the other samples are still 1, which can make the model pay more attention to the samples of the medium-grained label category and strengthen the learning of violation types, when Level = 3, the model needs to accurately locate the specific violation elements, and the sample y k belonging to the fine-grained label category is given a weight ω(y k , Level) = 2.0, and the other samples are 1, which makes the model focus more on the samples of the fine-grained label category and improves the identification ability of specific violation elements. Through this dynamic weight adjustment, the model can learn the features of key samples according to the label granularity and task requirements in different training stages, and improve the training effect.

[0048] Finally, enter the model iteration optimization stage (S500), input the to-be-audited image-text content into the trained multi-modal large model, the model analyzes and processes the input image-text content according to the knowledge and features learned in the multi-granularity contrast learning training stage, the model will first extract the visual features of the image and the semantic features of the text, then fuse these features through the attention mechanism, and then predict the violation categories of the image-text content according to the fused multi-modal features and learned violation patterns, output the image violation category, text violation category and image-text comprehensive violation judgment result, complete the image-text content review, in the actual application process, if it is found that the review effect of the model does not meet the expectation, new to-be-audited image-text content can be collected, and the new pseudo label is generated by re-entering the multi-modal pseudo label generation stage (S100), and the above training and optimization process is repeated to iteratively update the model, and the review accuracy and performance of the model are continuously improved to adapt to the changing image-text content violation forms and review requirements.

[0049] In summary, the embodiment constructs a complete image-text content review method based on a multi-modal large model through the steps of multi-modal pseudo-label generation, pseudo-label quality evaluation and screening, label system construction, multi-granularity contrast learning training, and model iterative optimization. The method filters high-quality pseudo-labels through multi-dimensional confidence evaluation indicators, ensuring the reliability of the training data. The constructed three-stage dynamic adjustment label system allows the model to gradually improve the review accuracy at different training stages, avoiding low training efficiency and overfitting problems. Combined with the contrast learning strategy and dynamic weight adjustment semi-supervised training method, the image and text features are effectively fused, improving the model's ability to identify integrated illegal content of images and texts. Through the method of this embodiment, the image-text content review task can be efficiently and accurately completed.

[0050] Embodiment Two

[0051] As shown in Figure 1 , the embodiment provides a workflow of the image-text content review method based on a multi-modal large model. The specific steps of the workflow are as follows:

[0052] Input unlabeled image-text data, use a pre-trained visual encoder to process the image part of the unlabeled image-text data, and extract the global and local features of the image;

[0053] Use a pre-trained text encoder to process the text part of the unlabeled image-text data, and extract the syntax features of the text;

[0054] Fuse the image features and text features through an attention mechanism to generate a comprehensive multi-modal feature representation;

[0055] Based on the fused multi-modal features, use the prediction layer of the multi-modal large model to generate an initial pseudo-label set, each pseudo-label including image violation categories, text violation categories, and integrated image-text violation judgment results;

[0056] Design a confidence evaluation indicator that considers the model's prediction certainty, feature similarity, and cross-modal consistency;

[0057] Score the quality of each pseudo-label based on the model's prediction probability, feature similarity with labeled data, and consistency of image and text violation categories;

[0058] Select pseudo-labels with a confidence score higher than 0.8 as a high-quality pseudo-label dataset for subsequent training;

[0059] Discard low-confidence pseudo-labels to ensure the reliability of the training data;

[0060] The high-quality pseudo-label dataset is divided into three levels: coarse-grained, medium-grained, and fine-grained.

[0061] In the early stage of training, coarse-grained labels are used to establish the basic classification ability of the model, and the accuracy of the model is monitored.

[0062] When the model accuracy reaches the first preset value and the accuracy fluctuation is small, switch to medium-grained labels to refine the violation type identification.

[0063] When the model accuracy reaches the second preset value and the accuracy is stable, switch to fine-grained labels to improve the accuracy of the audit.

[0064] According to the training round, accuracy, and fluctuation coefficient, dynamically adjust the label granularity, and feed back the current granularity level information to the training stage.

[0065] Collect the labeled sample dataset (artificial labeling) and high-quality pseudo-label dataset as training input.

[0066] Receive the current granularity level information (coarse, medium, or fine-grained) from the label system construction stage.

[0067] Extract image and text features for each sample and fuse them into a comprehensive feature vector.

[0068] Construct positive samples (samples of the same violation category as the current sample) and negative samples (samples of different violation categories from the current sample).

[0069] According to the current granularity level, assign sample weights: all samples have the same weight in coarse-grained, increase medium-grained sample weights in medium-grained, and increase fine-grained sample weights in fine-grained.

[0070] Use a contrast learning strategy for semi-supervised training to optimize the multi-modal large model and strengthen the association between image and text features.

[0071] Input the image-text content to be audited into the trained multi-modal large model.

[0072] The model automatically analyzes the image and text content based on the learned multi-modal features and knowledge.

[0073] Output the audit results, including violation categories (such as image violations, text violations, or image-text comprehensive violations) or compliance judgments.

[0074] Use the results in the actual content audit system to complete the audit process.

[0075] The above merely describes the preferred embodiments of the present application, and is not intended to limit the present application in any form. Although the present application has been disclosed with the preferred embodiments as above, it is not intended to limit the present application. Any person skilled in the art can make some changes or modifications to the above disclosed technical content to obtain equivalent embodiments with equivalent changes, as long as the changes or modifications do not deviate from the technical solution of the present application. Any modification, change, equivalent change and modification of the above embodiments made according to the technical essence of the present application still belong to the scope of the technical solution of the present application.

Claims

1. A method for image-text content review based on a multi-modal large model, characterized in that, The specific steps of this method are as follows: S100, Multimodal pseudo-label generation: Utilize a multimodal large model to generate pseudo-labels for unlabeled image and text data, and fuse image features and text features to generate an initial set of pseudo-labels; S200, Pseudo-label quality assessment and screening: Design confidence assessment metrics to score the quality of pseudo-labels, screen high-confidence pseudo-labels, and construct a high-quality pseudo-label dataset for subsequent training; S300, Label System Construction: Divide the S200 high-quality pseudo-label dataset into three levels and dynamically adjust the label granularity in stages; S400, Multi-granularity contrastive learning training: Utilizing the high-quality pseudo-label dataset and label system adjustment information of S200, combined with contrastive learning strategies, semi-supervised training is performed on a large multimodal model; The semi-supervised training steps are as follows: The labeled sample set is obtained Each sample in the set has a manually labeled label, which is used to provide supervision information for the model, and the high-quality pseudo-label data set is , as supplementary data of the labeled sample, receiving the label system adjustment information from S300, obtaining the current label granularity level For any sample in the labeled sample set , the visual feature vector of the image and the semantic feature vector of the text are obtained respectively, and the comprehensive feature vector is obtained after feature fusion, and the corresponding accurate label is For any sample in the high-quality pseudo-label data set , the comprehensive feature vector is also obtained, and the corresponding pseudo label is ; Using contrastive learning loss function Semi-supervised training is performed on a large multimodal model, and the loss function is... ,in, This indicates the number of samples included in each training batch. Indicates the first in the current batch One sample, This is the cosine similarity calculation function, used to measure the degree of similarity between two feature vectors. It is the first The feature vector of each sample Representatives and Samples The feature vectors of positive samples of the same category are simultaneously derived from [the following] when constructing positive samples. and Searching for and Label For the same sample, obtain its feature vector as , Representatives and Samples Feature vectors of negative samples from different categories, from and Filter out tags and For different samples, their feature vectors are obtained as negative sample feature vectors. , This indicates the number of negative samples, where m represents the index of the negative sample. Is with sample label and the current tag granularity level The relevant weighting coefficients, This is a hyperparameter with an initial value of 0.1, and its value is determined based on the tag granularity level. Dynamic adjustment: when hour, =0.1, when hour, =0.05, when hour, =0.03; S500, Model Iteration and Optimization: Input the text and image content to be reviewed into the trained multimodal large model. The model outputs the review results based on the learned knowledge and features, thus completing the review of the text and image content.

2. The method of claim 1, wherein the method is performed by a multi-modal large model-based image-text content review method. Specifically, S100 includes: The pre-trained visual encoder extracts features from unlabeled image and text data, extracting global and local features to form the image's visual feature vector. By using a pre-trained text encoder, features are extracted from unlabeled text data, including syntactic features, which are then used to construct semantic feature vectors. The visual feature vector of the image and the semantic feature vector of the text are weighted and fused through an attention mechanism. Based on the weighted and fused multimodal features, an initial set of pseudo-labels is generated through the prediction layer of a multimodal large model. Each pseudo-label in the initial set of pseudo-labels contains the image violation category, the text violation category, and the combined image and text violation judgment result.

3. The method for reviewing text and image content based on a multimodal large model according to claim 1, characterized in that, The S200 design confidence evaluation index, quality score of pseudo label: Wherein, represents the confidence score of the pseudo label , the value range is , is the probability value of the multi-modal large model predicting the pseudo label , the value range is , is the similarity of the pseudo label corresponding to the image-text feature and the annotated similar image-text feature, which is calculated by the cosine similarity formula, the value range is , the formula is: , is the image-text fusion feature vector corresponding to the pseudo label , is the fusion feature vector of the annotated similar image-text, represents the length of the vector, is the cross-modal consistency score of the image illegal category and the text illegal category in the pseudo label , the value range is , which is obtained by calculating the cosine similarity of the image and text illegal category feature vector, is a weight coefficient, satisfying , and is used to adjust the importance of different evaluation dimensions, and the pseudo label with a confidence score >0.8 is screened out to construct a high-quality pseudo label data set.

4. The method of claim 1, wherein the method further comprises: The three levels of S300 are: coarse-grained, medium-grained, and fine-grained, wherein: Coarse-grained label generation: For high-quality pseudo-label data, use a multimodal large model to predict the major violation category, which is then used as a coarse-grained high-confidence sample to be input into the multimodal large model to establish the model foundation, and the model accuracy is calculated. Medium-granularity label generation: For coarse-grained high-confidence samples, refine the predicted violation behavior and use it as a medium-granularity high-confidence sample. When the model's accuracy reaches the first preset value during contrastive learning training, introduce medium-granularity labels to refine the violation type. Fine-grained tag generation: For medium-grained high-confidence samples, specific violation elements are located. When the accuracy reaches the second preset value and tends to stabilize, fine-grained tags are used to improve the accuracy of the review.

5. The method for reviewing text and image content based on a multimodal large model according to claim 1, characterized in that, The S300 sets the current training round as , the accuracy of the model after the first round of training is , the first preset accuracy threshold is defined as , the second preset accuracy threshold is , and ; at the same time, the accuracy fluctuation coefficient is set to measure the stability of the accuracy, the accuracy , the and when =1, =0, wherein, represents the accuracy of the model after the first round of training, the value range is , represents the accuracy of the model after the first round of training, represents the number of image-text samples correctly predicted as non-compliant by the model after the first round of training, that is, the number of samples that are actually non-compliant and are also judged as non-compliant by the model, represents the number of image-text samples correctly predicted as compliant by the model after the first round of training, that is, the number of samples that are actually compliant and are also judged as compliant by the model, represents the number of image-text samples incorrectly predicted as non-compliant by the model after the first round of training, that is, the number of samples that are actually compliant but are misjudged as non-compliant by the model, represents the number of image-text samples incorrectly predicted as compliant by the model after the first round of training, that is, the number of samples that are actually non-compliant but are misjudged as compliant by the model, is the fluctuation coefficient of the accuracy in the first round of training, and the label granularity level is represented by , , wherein 1 represents coarse granularity, 2 represents medium granularity, and 3 represents fine granularity.

6. The method of claim 5, wherein the method further comprises: The S300 phased dynamic adjustment of label granularity specifically refers to: When , , , a base classification capability of the model is established using coarse-grained labels, is the number of initial training rounds set. When and then: , introduce medium-grained labels, refine violation type identification, is the stable threshold for medium-grained switching and = 0.05, and when , maintain ; when and hour: Using fine-grained tags improves the accuracy of the review process, and when Then maintain , The stable threshold for fine-grained switching and ; After each adjustment of the tag granularity level, the current... Information regarding adjustments to the labeling system is fed back to S400.

7. The method for reviewing text and image content based on a multimodal large model according to claim 1, characterized in that, During the training of a multimodal large model, the S300 adjusts the current label granularity level. Assign different weights to the samples: when When the weight is 1, all samples are assigned the same weight, meaning each sample contributes equally to the calculation of the loss function. ; when When =2, If it belongs to the medium-granularity label category, then 1.5, otherwise 1; when hour, If it belongs to the fine-grained tag category, then 2.0, otherwise 1.

Images