Material identification method based on multi-modal large model

Abstract

The invention discloses a material identification method based on a multi-modal large model, and particularly relates to the technical field of data processing. The method comprises the following steps: inputting image data and associated text data of advertisement materials into a multi-modal large model, and carrying out preliminary preprocessing and feature extraction; respectively extracting visual features and semantic features of the image and the text by using a convolutional neural network and a natural language processing technology, and optimizing a fusion effect of the image and text features through a multi-modal contrast learning strategy; the feature alignment condition is verified by comparing a loss function, and the recognition precision of the model is improved by combining a double-path fine adjustment mechanism and multi-label classification supervision; finally, an improved cue word mechanism is adopted to carry out accurate classification on the advertisement materials, a putting engine and a feedback mechanism are combined, the auditing effect of the advertisement materials is monitored and optimized in real time, the accuracy of advertisement compliance management is remarkably improved, auditing errors are reduced, and the customer satisfaction degree is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a material recognition method based on a multimodal large model. Background Art

[0002] With the diversification and complexity of advertising, compliance review and content understanding of advertising materials have become important links in media platform access. Different types of advertising materials have significant differences in form, content and presentation, making it difficult for traditional review systems to meet real-time, strict and high-precision recognition requirements. Although the existing multimodal large models have general image and text understanding capabilities, when dealing with game, click-bait and skit advertisements, the recognition accuracy is often low due to the disconnection between image content and advertising intent. In addition, the current material review system has problems of lag and generalization in terms of blocking, feedback and optimization, which affects the stability of advertising delivery and even causes interruptions in cooperation between media platforms and advertisers and customer complaints.

[0003] The prior art has the following deficiencies: The problem with existing technologies is that large multimodal models have difficulty accurately understanding complex advertising materials. For example, in game ads, the images displayed are often unrelated to the actual game scenes and may contain elements such as induced clicks and fake pop-ups, making it impossible for the review system to judge their true attributes based on image information alone. Similarly, in skit ads, the correlation between image content and text information is weak, and existing models find it difficult to effectively integrate the semantics of the two, resulting in a high rate of misjudgment in reviews. These problems not only affect the compliance management of advertising, but may also lead to a decline in user experience, further exacerbating the instability of advertising delivery. Summary of the invention

[0004] The purpose of the present invention is to provide a material recognition method based on a multimodal large model to address the deficiencies in the background technology.

[0005] In order to achieve the above object, the present invention provides the following technical solution: a material recognition method based on a multimodal large model, comprising: By inputting the image data and associated text data of the advertising material, it is passed to the multimodal large model for preliminary preprocessing; The multimodal large model is used to extract the visual features of image data and the semantic features of text data respectively, and the image and text are represented by convolutional neural network and natural language processing technology to obtain multi-dimensional feature vectors. Through a multimodal contrastive learning strategy, image features and text features are fused and contrastive learning is performed, including judging whether image and text features are aligned in a shared space through a contrastive loss function; Through the dual-path fine-tuning mechanism, combined with contrastive learning and multi-label classification supervision, the recognition ability of the model is optimized. Based on the optimized large model, the improved prompt word mechanism is used to classify the advertising materials. Based on the classification results, the delivery engine is used to further review the advertising materials, and the feedback mechanism is used to monitor the material review effect in real time to optimize the classification effect and accuracy of the model.

[0006] Preferably, the image data and associated text data of the input advertising material are passed to the multimodal large model for preliminary preprocessing, and the preprocessing process includes resizing and normalizing the image data, and cleaning and segmenting the text data.

[0007] Preferably, visual features and semantic features are extracted from image data and text data respectively, and a contrast loss function is constructed. The loss function enhances the similarity between the image and the related text by minimizing the distance between the positive sample and the negative sample. The loss function expression is: ;in: is the label of the sample pair, which is 1 if the image and text are matched, otherwise 0; is the distance between image and text features in the shared space, measured using Euclidean distance or cosine similarity, and m is a margin value used to control the minimum distance between positive and negative samples; During the training process, the system generates a large number of positive and negative sample pairs randomly or through data sampling. Each sample pair consists of an image and a text. The positive sample pairs contain related images and texts, while the negative sample pairs contain unrelated images and texts. For each pair of image and text, the similarity in the shared space is calculated to determine whether they are aligned. For the positive sample pair, the obtained similarity is compared with the similarity threshold. If the similarity is greater than the similarity threshold, it means that the image and text description are highly correlated. For the negative sample pair, the obtained similarity is compared with the similarity threshold. If the similarity is less than the similarity threshold, it means that the image and text description are irrelevant.

[0008] Preferably, the features of the image and the text are fused to generate a high-dimensional multimodal feature vector, and the advertising material is classified into multiple labels based on the multimodal feature vector.

[0009] Preferably, the platform conducts real-time evaluation of the review effect of the advertising material based on the classification results of the advertising material and user behavior data, and makes adjustments and optimizations.

[0010] Preferably, the conversion correlation anomaly index is generated after analyzing the correlation between the advertisement display and the user purchase conversion, wherein the conversion correlation anomaly index is obtained by: set up Data representing the number of times an ad was displayed; represents the corresponding number of conversions, and n is the number of time periods; calculate the correlation value between the number of ad impressions and the number of conversions , the calculation formula is: ;in: and is the data of ad impressions and conversions in the i-th time period, and are the mean of ad impressions and conversions respectively; linear regression is used to predict the conversion value for each time period , calculate the residual , the residual is the difference between the actual conversion and the predicted conversion: ;in: is the actual number of conversions; the residual is standardized and the calculation formula is: ;in: is the standard deviation of all residuals; calculate the transformation association anomaly index: ; In the formula, is the conversion-related anomaly index.

[0011] Preferably, the emotional feedback of the user to the advertising material is analyzed to generate an emotional feedback fluctuation index, wherein the emotional feedback fluctuation index is obtained by: Collect emotional feedback data of advertising materials, select neighborhood size K, and build an emotional feedback dataset , where each Represents the emotional feedback in a certain period of time; for each data point , calculate the distance between it and the Kth nearest neighbor , the expression is: ; Where: p and o are two data points; is the distance from point o to its Kth nearest neighbor; d(p,o) is the actual distance between point p and point o; for each data point, calculate its local reachability density , that is, calculate the density of point p and other points in its neighborhood. The formula is: ;in: are the K nearest neighbors of data point p; for each data point p, calculate the ratio of the density of point p to the density of its neighborhood , the formula is: ;in: is the local reachable density of point o. The LOF value is calculated for all the emotional feedback data points of the advertising materials. The points with LOF values ​​higher than a certain threshold are regarded as abnormal fluctuations. The emotional feedback fluctuation index is calculated. The expression is: ; In the formula, It is the sentiment feedback fluctuation index.

[0012] Preferably, the conversion-related anomaly index and the emotional feedback fluctuation index are normalized so that they are both between [0, 1], and the audit effect analysis factor of the material is calculated based on the normalized conversion-related anomaly index and the emotional feedback fluctuation index.

[0013] Preferably, the audit effect analysis factor of the acquired material is compared with the audit effect analysis factor reference value preset based on historical data. If the audit effect analysis factor of the material is greater than or equal to the preset audit effect analysis factor reference value, it means that the audit effect of the material is excellent; if the audit effect analysis factor of the material is less than the preset audit effect analysis factor reference value, it means that the audit effect of the material is poor.

[0014] In the above technical solution, the technical effects and advantages provided by the present invention are: The present invention utilizes a multimodal large model combined with convolutional neural networks and natural language processing technology to extract features of images and texts, and uses contrastive learning and dual-path fine-tuning mechanisms to enhance model recognition accuracy. At the same time, the conversion correlation anomaly index and the sentiment feedback fluctuation index are combined to conduct real-time evaluation of advertising materials, and the audit effect and compliance management of materials are further optimized through normalization processing and calculation of audit effect analysis factors, thereby improving the quality of advertising materials, reducing audit errors, improving customer satisfaction, and enhancing the accuracy and effect of advertising delivery. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0016] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] For examples, see Figure 1 As shown, the material recognition method based on a multimodal large model described in this embodiment includes: By inputting the image data and associated text data of the advertising material, it is passed to the multimodal large model for preliminary preprocessing; The multimodal large model is used to extract the visual features of image data and the semantic features of text data respectively, and the image and text are represented by convolutional neural network and natural language processing technology to obtain multi-dimensional feature vectors. Through a multimodal contrastive learning strategy, image features and text features are fused and contrastive learning is performed, including judging whether image and text features are aligned in a shared space through a contrastive loss function; Through the dual-path fine-tuning mechanism, combined with contrastive learning and multi-label classification supervision, the recognition ability of the model is optimized. Based on the optimized large model, the improved prompt word mechanism is used to classify the advertising materials. Based on the classification results, the delivery engine is used to further review the advertising materials, and the feedback mechanism is used to monitor the material review effect in real time to optimize the classification effect and accuracy of the model.

[0019] In the process of inputting and preprocessing the creative image data and associated text data, and passing them to the multimodal large model for preliminary preprocessing, the specific steps are as follows: 1. Preprocessing of image data: Resizing: In order to make the image data suitable for the input requirements of the multimodal large model, the image is first resized. Usually, the image is resized to a fixed size (such as 224x224 pixels or other sizes required by the model) to ensure that the image can be input into the convolutional neural network (CNN) for feature extraction.

[0020] Normalization: Normalize the pixel values ​​of the image and scale the pixel value range of the image from the original [0,255] to [0,1] to improve the convergence speed during model training and reduce the instability of numerical calculations. In addition, color standardization may also be performed to make the color distribution of the image more balanced.

[0021] 2. Preprocessing of text data: Text cleaning: Clean the text data, remove irrelevant information and noise (such as special symbols and HTML tags in advertisements), and retain the core language information. This step helps reduce redundant content in the model input.

[0022] Word segmentation and word vectorization: The cleaned text is segmented to split the sentence into individual words or subwords. Then a pre-trained word vector model (such as Word2Vec, GloVe, or BERT) is used to convert each word into a fixed-length vector representation to capture the semantic information of the vocabulary.

[0023] Text encoding: For longer texts, an encoder (such as Transformer models such as BERT and GPT) is used to convert the text into a high-dimensional semantic representation so that it can be input into the subsequent multimodal model. For short texts, a simple bag-of-words model or TF-IDF may be used as a feature representation.

[0024] 3. Unified processing of multimodal input: Aligning image and text feature spaces: The preprocessing results of images and text (image features and text vectors) are fed into a large multimodal model. Usually, the feature spaces of images and texts are different, so a special alignment method is needed to map them to the same shared feature space. For example, feature vectors are extracted from image data through CNN, and semantic vectors are extracted from text data through models such as LSTM and Transformer. Then, the image features and text features are concatenated or weightedly fused to form a joint feature representation.

[0025] 4. Feature extraction and transfer to multimodal model: Multimodal fusion: The pre-processed image features and text features are fused through a large multimodal model. Common fusion methods include early fusion (directly merging image and text features at the input layer), mid-term fusion (fusion at the middle layer of the network), and late fusion (fusion at the output layer of the model). In this way, image and text data can complement and promote each other, improving the overall understanding and recognition capabilities of advertising materials.

[0026] Output shared space representation: Through this fusion, the multimodal information of images and text is converted into a shared feature space, which serves as the basis for subsequent classification, detection, and analysis tasks.

[0027] In multimodal feature extraction and representation, the multimodal large model is used to extract the visual features of image data and the semantic features of text data respectively. The image and text are represented by convolutional neural network (CNN) and natural language processing (NLP) technology. The process of obtaining multidimensional feature vectors can be divided into the following detailed steps: 1. Visual feature extraction of image data (using convolutional neural networks): Input image: First, input the preprocessed image data into the convolutional neural network (CNN). CNN is the core network architecture for image processing. It extracts local and global features of the image through stacked convolutional layers, pooling layers, and fully connected layers. Common CNN models such as ResNet, VGG, Inception, etc. can be used as the basic network for feature extraction.

[0028] Convolutional layer feature extraction: In the convolutional layer, the image is scanned by convolution kernels of different sizes to extract local features of the image (such as edges, textures, etc.), and this information is gradually aggregated into a higher-level feature representation.

[0029] Pooling layer dimensionality reduction: Pooling layers (such as max pooling or average pooling) are used to reduce the dimensionality of feature maps, retaining important features while reducing computational complexity.

[0030] Fully connected layer representation: In the last layer of CNN, there is usually a fully connected layer that converts the extracted local features into global features and finally outputs a high-dimensional visual feature vector. This vector contains the overall information of the image and provides a basis for subsequent multimodal feature fusion.

[0031] 2. Semantic feature extraction of text data (using natural language processing technology): Input text: The preprocessed text is input into the natural language processing (NLP) model, usually using pre-trained language models such as BERT, GPT, LSTM, etc. These models can effectively extract semantic features in the text.

[0032] Word segmentation and embedding: The text is first segmented to divide the sentence into words or subwords, and then each word is converted into a word vector through word embedding (such as Word2Vec, GloVe or BERT's embedding layer). Each word vector captures the grammatical and semantic information of the word.

[0033] Text encoding: Use language models (such as BERT, GPT) for context encoding, which can capture the relationship and contextual information between words in the text. In this process, the model gradually builds a deep semantic representation of the text through the self-attention mechanism, so that the embedding of each word or subword can take into account the context of the entire text.

[0034] Extract sentence-level features: Finally, after being processed by the encoder, the text will be converted into a high-dimensional semantic feature vector, which contains the semantic information of the entire text, and the context and long-range dependencies of the text have been considered during the processing.

[0035] 3. Fusion of visual features and semantic features: Feature alignment and splicing: After the image and text are extracted by CNN and NLP models respectively, we get two feature vectors of different modalities (visual feature vector and text feature vector). In order to merge these two features, the following methods are usually used: Early fusion: directly concatenate image features and text features into a larger vector as input to the model.

[0036] Mid-term fusion: The image and text features are interacted in the intermediate layer, usually using a multimodal neural network (such as attention mechanism, bidirectional LSTM, etc.) to fuse image and text features.

[0037] Late fusion: Process images and text separately, then fuse their outputs for the final decision or classification.

[0038] Shared feature space mapping: Mapping visual features and text features to a common feature space through contrastive learning or other alignment methods can better understand the relationship between images and text. For example, using the contrastive loss function to minimize the distance between feature vectors of similar materials and maximize the distance between unrelated materials.

[0039] 4. Generate multi-dimensional feature vectors: Output high-dimensional vector: After the image and text features are fused, a high-dimensional multimodal feature vector will be generated. This vector contains both the visual information of the image and the semantic information of the text, and can express the correlation between the two.

[0040] Subsequent processing: This multidimensional feature vector can be used as input for subsequent classification, regression, recommendation or generation tasks to further improve the understanding and judgment of advertising materials. For example, it can be fed into a classifier to determine the type of advertisement, or a regression model can be used to predict the performance of advertising materials.

[0041] In the multimodal contrastive learning strategy, image features and text features are fused and their alignment in the shared space is determined through contrastive learning.

[0042] First, through the above steps, visual features and semantic features are extracted from image data and text data respectively. Image features are extracted by convolutional neural network (CNN), and text features are extracted by natural language processing technology (such as BERT, GPT).

[0043] In order to be able to compare effectively, image features and text features need to be mapped to the same shared feature space. In this process, a shared neural network layer or a specific mapping function is usually used to transform the features of the two modalities so that they can be compared in the same space. Fully connected layers, alignment mapping networks, or other transformation methods can be used to achieve this goal.

[0044] In order to align image and text features in a shared space, a contrast loss function needs to be constructed. This loss function enhances the similarity between the image and the related text by minimizing the distance between the positive sample and the negative sample, while increasing the distance between the image and the irrelevant text. The loss function expression is: ;in: is the label of the sample pair, which is 1 if the image and text are matched, otherwise 0; It is the distance between image and text features in the shared space, usually measured using Euclidean distance or cosine similarity. m is a marginal value used to control the minimum distance between positive and negative samples.

[0045] Positive samples: Positive samples refer to images and their corresponding text descriptions (or texts that are highly related to them). Such samples should have a small distance in the shared space.

[0046] Negative samples: Negative samples refer to images with irrelevant text descriptions or texts with completely different semantics. The distance of such samples should be enlarged to ensure that they are located farther away in the shared space.

[0047] During the training process, the system generates a large number of positive and negative sample pairs randomly or through data sampling. Each sample pair consists of an image and a text, where the positive sample pair contains related images and texts, and the negative sample pair contains unrelated images and texts.

[0048] For each pair of image and text, we determine whether they are aligned by calculating their similarity in the shared space (such as through cosine similarity or Euclidean distance). The purpose of similarity calculation is to measure the "closeness" of the image and text in the common feature space. For positive sample pairs, the obtained similarity is compared with the similarity threshold. If the similarity is greater than the similarity threshold, it means that the image and text description are highly correlated; for negative sample pairs, the obtained similarity is compared with the similarity threshold. If the similarity is less than the similarity threshold, it means that the image and text description are unrelated.

[0049] By minimizing the contrastive loss function, the model can effectively adjust its parameters so that images and related text are closer in the shared space, while the distance between images and unrelated text is increased. During the training process, the model gradually learns more effective feature representations, thereby improving the cross-modal alignment effect.

[0050] The goal of optimization is to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs, so that the representation of images and text in the shared space is more aligned. This process is completed through the back propagation algorithm, and the model parameters are gradually updated through gradient descent, ultimately achieving the best alignment of image and text features in the common embedding space.

[0051] After multiple rounds of comparative learning, image features and text features will have higher semantic consistency. The representation of images and text in a shared space will be able to effectively reflect the semantic relevance between them and support subsequent tasks (such as advertising material classification, recommendation, label generation, etc.).

[0052] After the contrastive learning optimization is completed, the features of the image and text will be fused into a unified multimodal feature vector, which can represent the semantic content of both the image and the text and effectively reflect the semantic association between the two. This multimodal feature vector will be used as input in subsequent tasks (such as classification, recommendation, effect prediction, etc. of advertising materials).

[0053] The dual-path fine-tuning mechanism combines contrastive learning with multi-label classification supervision, and is mainly used to improve the performance of multimodal large models in advertising material recognition. The following are the specific optimization steps: Collect a large amount of advertising materials, including images and related text information (such as advertising copy, labels, product descriptions, etc.).

[0054] Image data: standardization processing, such as resizing, color space conversion, etc. Text data: text preprocessing, cleaning stop words, performing word segmentation or embedding, and converting to a format suitable for input models.

[0055] Use convolutional neural networks (CNN) or deep convolutional networks (such as ResNet, EfficientNet) to extract visual features of images. Embed image features into a shared space. Use pre-trained language models (such as BERT, RoBERTa) to extract semantic features of text. Text features are embedded and mapped to the same space as image features.

[0056] Compute the similarity between images and texts, maximizing the similarity between related pairs (images and texts of the same creative) through contrastive learning, while minimizing the similarity between unrelated pairs (images and texts of different creatives). Use Euclidean distance (or cosine similarity) to measure the distance between image and text features.

[0057] Define multiple category labels for each creative. For example: main category (such as financial ads, game ads, click-through ads, etc.), style category (such as realistic, cartoon, abstract, etc.), element category (such as containing currency symbols, characters, etc.); Loss function: train the network through multi-label classification loss (such as cross entropy loss) to optimize the multi-label classification task of images and text. Combine contrastive learning and multi-label classification supervision to optimize the model through a joint loss function. The loss function includes: Contrastive loss: maximize the alignment between image and text features through contrastive learning. Multi-label classification loss: optimize the accurate classification of creatives through multi-label classification.

[0058] Use the training data with the above joint loss to train the model. Use an optimization algorithm (such as Adam or SGD) to perform gradient descent optimization to adjust the weights of the network so that the model can achieve better alignment between images and text. Adjust hyperparameters such as learning rate and batch size to ensure that the training process converges.

[0059] The steps of ad creative classification based on the optimized large model include: First, the image data and associated text data of the advertising material are input into the optimized multimodal large model. The image data may include the visual content of the advertisement, such as images, video frames, etc., while the text data is the copy, labels, descriptions and other information in the advertisement.

[0060] The model processes input data through two independent paths: Visual path: The model processes the ad image through a convolutional neural network (CNN) to extract visual features from the image. These features include elements such as the image's structure, color, and shape. Text path: At the same time, the model processes the ad text through natural language processing techniques (such as BERT, RoBERTa, etc.) to extract semantic features of the text. These features reflect the meaning and grammatical structure of the text content.

[0061] After image and text feature extraction, the model will perform contrastive learning. Through contrastive learning strategy, the model will measure the degree of alignment of image features and text features in a shared space. In other words, the model will determine whether the image and text are part of the same ad creative, ensuring that the semantics between the image and text are consistent. If the similarity between image and text features is high, it means that they belong to the same type of ad; otherwise, it indicates that there are large differences between them and they may be different types of ad creatives.

[0062] After contrastive learning, the model will fuse the features of the image and text into a common feature space. Through multimodal contrastive learning, the features of the image and text are jointly optimized to achieve more accurate semantic matching. This process ensures that the features between the image and text are aligned in the same semantic space.

[0063] After the image and text features are fused, the model will enter the classification stage of the advertising material. Through multi-label classification, the model classifies the advertising material according to different features, such as: Ad type classification: The model will identify the type of advertisement, such as financial advertisement, game advertisement, skit advertisement, etc.

[0064] Ad content classification: Based on the visual elements and textual content of the ad, the model may further classify the ad’s style, content type (e.g., entertainment, education, product promotion, etc.).

[0065] Each ad creative may be assigned multiple labels, which represent multiple attributes of the ad and meet the requirements of multi-label classification.

[0066] After completing the classification task, the model will generate classification results for each creative. These results will express the probability of the creative belonging to different categories in the form of correlation coefficients. For example, a creative may have a correlation coefficient of 0.7 in the "Game Ads" category and a correlation coefficient of 0.3 in the "Click-through" category.

[0067] Once the classification results are generated, further processing can be performed based on these results. For example: Filtering non-compliant ads: If an ad material is classified as a click-bait ad or an ad type that violates the platform's regulations, it can be filtered or marked for review. Ad delivery optimization: Based on the ad classification results, the platform can optimize the ad delivery strategy, select the appropriate ad type, delivery channel, and target audience, and improve the effectiveness and efficiency of ad delivery.

[0068] Based on the classification results, the delivery engine is used to further review the advertising materials, and the specific steps for real-time monitoring of the material review results through the feedback mechanism are as follows: After the advertising materials are classified by the multimodal large model, the system will conduct a preliminary review based on the classification results of each material (such as financial ads, game ads, click-induced ads, etc.). At this time, the system will filter out advertising materials that do not meet the regulations based on existing rules or prohibition conditions. For advertising materials that do not meet platform policies or user needs (such as click-induced, overly intrusive ads, ads with malware, etc.), the delivery engine will directly mark them as unqualified, filter them, and refuse to deliver them.

[0069] The system uses the classification results to determine whether the advertising material meets the preliminary standards of the platform. For example, if it is classified as an "inducing click" ad, the system will directly mark it as a violation and prohibit its delivery. At this time, the system can send a rejection notice to the advertiser and prompt it to change the material. For advertising materials that are not marked as violations, the delivery engine will further refine the review process based on the category, content, and form of the advertising material. For example, for "game ads", the system may check whether the material complies with the specific regulations of game ads, such as whether there is misleading content, false propaganda, etc. For complex or suspected illegal advertising materials, the delivery engine will assign them to high-precision partitions, and further use advanced models for in-depth analysis to ensure that the material meets the platform requirements in multiple dimensions.

[0070] After the creative review is completed, the platform will return the review results to the advertiser and provide feedback. If the creative is judged to be qualified, it will enter the delivery process; if the creative is judged to be unqualified, the advertiser will be required to modify or resubmit. This feedback process is real-time, ensuring that advertisers are aware of the review status in a timely manner.

[0071] The system also regularly collects feedback data from media platforms and users, such as the click-through rate of advertising materials, customer complaint rate, delivery effect, etc. These data are crucial for the subsequent optimization of the classification accuracy of the large model.

[0072] Through real-time monitoring and user feedback, the system can regularly fine-tune the model. For example, if it is found that the classification accuracy of a certain type of advertising material is low (such as skit ads or click-bait ads), the system will use new labeled data to retrain the large model to optimize the model's classification effect.

[0073] Based on the feedback mechanism and user behavior data, the model will perform incremental learning to continuously improve the ability to identify and classify advertising materials. For example, as the platform continues to receive new advertising materials, the model will automatically update to improve the recognition accuracy of new advertising materials (such as new types of clickbait, skit ads, etc.).

[0074] The platform conducts real-time evaluation of the review effect of the creatives based on the classification results of the creatives and user behavior data, and makes adjustments and optimizations. The specific steps include: The system first classifies the advertising materials, for example, by analyzing the categories of advertising content through technologies such as image recognition and natural language processing. Possible categories include text ads, image ads, video ads, etc. Some advertising materials may show a high conversion rate, but they are only effective in specific user groups. After analyzing the correlation between ad display and user purchase conversion, the conversion correlation anomaly index is generated. The conversion correlation anomaly index is obtained as follows: Prepare the data set: Ad display data: ad display count (e.g., ad impressions, ad exposures, etc.) Conversion data: conversion count (e.g., purchases, post-click purchases, target behaviors, etc.) Time period: data can be aggregated by time units such as day, week, or month.

[0075] set up Data representing the number of times an ad was displayed (number of impressions per time period); represents the corresponding number of conversions (purchase conversion volume in each time period), and n is the number of time periods.

[0076] Calculate the correlation between ad impressions and conversions , the calculation formula is: ;in: and is the data of ad impressions and conversions in the i-th time period, and are the means of ad impressions and conversions, respectively.

[0077] Once the correlation value is calculated , linear regression can be used to predict the conversion value of each time period. The linear regression equation is: ;in: is the predicted transformation value, α and β are the intercept and slope of the regression equation, which can be calculated by the least squares method; calculate the residual , the residual is the difference between the actual conversion and the predicted conversion: ;in: is the actual number of conversions.

[0078] In order to determine some abnormal transformation points, it is necessary to standardize the residuals to facilitate the identification of data points with large deviations. The formula for calculating the standardized residuals is: ;in: is the standard deviation of all residuals; according to the standardized residual , set a threshold to identify outliers. If the absolute value of the standardized residual exceeds a certain threshold (for example, 2 or 3), the conversion data for that time period is considered abnormal. Calculate the conversion correlation anomaly index: ; In the formula, It is the conversion association anomaly index. If the CAI is large, it means that there is a large anomaly in the relationship between ad display and conversion. If the CAI is small, it means that the relationship between ad display and conversion is relatively normal.

[0079] The platform collects user behavior data, which may include click-through rate, interaction rate, length of stay, etc. The user's click, browse, skip or purchase behavior will directly affect the evaluation of the advertising effect; the emotional feedback fluctuation index is generated after analyzing the emotional feedback of the user on the advertising material. The method for obtaining the emotional feedback fluctuation index is as follows: It is necessary to collect sentiment feedback data for ad creatives, including user comments or feedback sentiment scores (e.g., sentiment scores from -1 to 1, or sentiment ratings from 0 to 5). User interactions with ad creatives (such as likes, shares, reposts, etc.) can be converted into sentiment feedback ratings.

[0080] Choose an appropriate neighborhood size K. The value of K needs to be selected according to the distribution of the data. A value of K that is too small may lead to oversensitivity, while a value of K that is too large may lead to overly broad detection.

[0081] The k-distance of each data point represents the distance from the point to its Kth nearest neighbor, and establishes a sentiment feedback dataset , where each Represents the emotional feedback in a certain period of time. For each data point , calculate the distance between it and the Kth nearest neighbor , the expression is: ; Where: p and o are two data points; is the distance from point o to its Kth nearest neighbor; d(p,o) is the actual distance between point p and point o; for each data point, calculate its local reachability density , that is, calculate the density of point p and other points in its neighborhood. The formula is: ;in: are the K nearest neighbors of data point p; for each data point p, calculate the ratio of the density of point p to the density of its neighborhood , the formula is: ;in: is the local reachable density of point o. If the LOF value of a data point is much greater than 1, it means that the density difference between the point and its neighborhood is large, and it may be an outlier. A lower LOF value (close to 1) means that the density of the point is less different from that of the neighborhood, and it is usually considered a normal point.

[0082] Calculate the LOF value for all the emotional feedback data points of the advertising material, and regard the points with LOF values ​​higher than a certain threshold as abnormal fluctuations, reflecting that the emotional feedback of the advertising material has significant fluctuations. Calculate the emotional feedback fluctuation index, the expression is: ; In the formula, It is the sentiment feedback fluctuation index.

[0083] The conversion correlation anomaly index and the emotional feedback fluctuation index are normalized so that they are both between [0,1]. The audit effect analysis factor of the material is calculated based on the normalized conversion correlation anomaly index and the emotional feedback fluctuation index.

[0084] For example, the present invention can use the following formula to calculate the audit effect analysis factor of the material, and the calculation expression is: ; In the formula, To analyze the audit effect of multiple materials, is the conversion association anomaly index, is the sentiment feedback fluctuation index, is the weight coefficient for transforming the association anomaly index and the emotional feedback fluctuation index (which can be optimized based on experimental experience or machine learning), and Both are greater than 0.

[0085] Compare the audit effect analysis factor of the acquired material with the audit effect analysis factor reference value preset based on historical data. If the audit effect analysis factor of the material is greater than or equal to the preset audit effect analysis factor reference value, it means that the audit effect of the material is excellent; if the audit effect analysis factor of the material is less than the preset audit effect analysis factor reference value, it means that the audit effect of the material is poor.

[0086] It should be noted here that the reference value of the audit effect analysis factor can be continuously adjusted and optimized through in-depth analysis of historical data and combined with the actual performance of the advertising materials to ensure that it can accurately reflect the audit effect of the materials.

[0087] Based on the feedback mechanism and user behavior data, the model will perform incremental learning and continuously improve the recognition and classification capabilities of advertising materials. For example, as the platform continues to receive new advertising materials, the model will be automatically updated to improve the recognition accuracy of new advertising materials (such as new types of induced clicks, short drama ads, etc.). According to the review history of advertising materials and platform feedback, the system can also adjust the review rules and strategies. For example, if a certain type of advertising material is repeatedly marked as qualified, but the actual performance is poor, the platform can adjust the weight parameters in the algorithm or add new review dimensions (such as the click-induced nature of the advertisement, the authenticity of the advertising content, etc.) to further optimize the classification. Through comprehensive analysis of the review and classification results, the platform can optimize the advertising delivery strategy. Based on the classification results of the advertising materials, the system can select the best delivery channel, delivery time, target audience, etc., thereby improving the effectiveness and accuracy of advertising delivery.

[0088] For certain creatives, the delivery engine can dynamically adjust the delivery of ads based on real-time monitoring data. For example, if a creative performs poorly after the initial delivery, the system will adjust its delivery strategy based on real-time feedback, which may include reducing the exposure of the creative or suspending delivery. Through the cyclic operation of the feedback mechanism, the review and delivery process of advertising creatives will continue to be optimized. Based on user experience data, the platform can further adjust the advertising classification model, review rules, and delivery strategy to ensure the compliance, effectiveness, and accuracy of the advertising. Ultimately, through this real-time monitoring and feedback mechanism, the platform can efficiently process advertising creatives, improve the accuracy and timeliness of advertising review, reduce misjudgments and missed judgments, and improve customer satisfaction and trust in media platforms.

[0089] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0090] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0091] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the objects associated with each other are in an "or" relationship, but it may also indicate an "and / or" relationship, which can be understood by referring to the context. A person of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this article can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0092] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A material recognition method based on a multimodal large model, characterized in that: include: By inputting the image data and associated text data of the advertising material, it is passed to the multimodal large model for preliminary preprocessing; The multimodal large model is used to extract the visual features of image data and the semantic features of text data respectively, and the image and text are represented by convolutional neural network and natural language processing technology to obtain multi-dimensional feature vectors. Through a multimodal contrastive learning strategy, image features and text features are fused and contrastive learning is performed, including judging whether image and text features are aligned in a shared space through a contrastive loss function; Through the dual-path fine-tuning mechanism, combined with contrastive learning and multi-label classification supervision, the recognition ability of the model is optimized. Based on the optimized large model, the improved prompt word mechanism is used to classify the advertising materials. Based on the classification results, the delivery engine is used to further review the advertising materials, and the feedback mechanism is used to monitor the material review effect in real time to optimize the classification effect and accuracy of the model.

2. The material recognition method based on a multimodal large model according to claim 1, characterized in that: The image data and associated text data of the input advertising material are passed to the multimodal large model for preliminary preprocessing. The preprocessing process includes resizing and normalizing the image data, and cleaning and segmenting the text data.

3. The material recognition method based on a multimodal large model according to claim 1, characterized in that: Visual features and semantic features are extracted from image data and text data respectively, and a contrast loss function is constructed. The loss function enhances the similarity between the image and the related text by minimizing the distance between the positive sample and the negative sample. The loss function expression is: ;in: is the label of the sample pair, which is 1 if the image and text are matched, otherwise 0; is the distance between image and text features in the shared space, measured using Euclidean distance or cosine similarity, and m is a margin value used to control the minimum distance between positive and negative samples; During the training process, the system generates a large number of positive and negative sample pairs randomly or through data sampling. Each sample pair consists of an image and a text. The positive sample pairs contain related images and texts, while the negative sample pairs contain unrelated images and texts. For each pair of image and text, the similarity in the shared space is calculated to determine whether they are aligned. For the positive sample pair, the obtained similarity is compared with the similarity threshold. If the similarity is greater than the similarity threshold, it means that the image and text description are highly correlated. For the negative sample pair, the obtained similarity is compared with the similarity threshold. If the similarity is less than the similarity threshold, it means that the image and text description are irrelevant.

4. The material recognition method based on a multimodal large model according to claim 1, characterized in that: The features of the image and text are fused to generate a high-dimensional multimodal feature vector, and multi-label classification is performed on the advertising material based on the multimodal feature vector.

5. The material recognition method based on a multimodal large model according to claim 4 is characterized in that: The platform conducts real-time evaluation of the review effect of the advertising materials based on the classification results of the advertising materials and user behavior data, and makes adjustments and optimizations.

6. The material recognition method based on a multimodal large model according to claim 5, characterized in that: After analyzing the correlation between ad display and user purchase conversion, a conversion correlation anomaly index is generated. The conversion correlation anomaly index is obtained as follows: set up Data representing the number of times an ad was displayed; represents the corresponding number of conversions, and n is the number of time periods; calculate the correlation value between the number of ad impressions and the number of conversions , the calculation formula is: ;in: and is the data of ad impressions and conversions in the i-th time period, and are the mean of ad impressions and conversions respectively; linear regression is used to predict the conversion value for each time period , calculate the residual , the residual is the difference between the actual conversion and the predicted conversion: ;in: is the actual number of conversions; the residual is standardized and the calculation formula is: ;in: is the standard deviation of all residuals; calculate the transformation association anomaly index: ; In the formula, is the conversion-related anomaly index.

7. The material recognition method based on a multimodal large model according to claim 6 is characterized in that: After analyzing the emotional feedback of the user on the advertising material, an emotional feedback fluctuation index is generated, wherein the method for obtaining the emotional feedback fluctuation index is as follows: Collect emotional feedback data of advertising materials, select neighborhood size K, and build an emotional feedback dataset , where each Represents the emotional feedback in a certain period of time; for each data point , calculate the distance between it and the Kth nearest neighbor , the expression is: ; Where: p and o are two data points; is the distance from point o to its Kth nearest neighbor; d(p,o) is the actual distance between point p and point o; for each data point, calculate its local reachability density , that is, calculate the density of point p and other points in its neighborhood. The formula is: ;in: are the K nearest neighbors of data point p; for each data point p, calculate the ratio of the density of point p to the density of its neighborhood , the formula is: ;in: is the local reachable density of point o. The LOF value is calculated for all the emotional feedback data points of the advertising materials. The points with LOF values ​​higher than a certain threshold are regarded as abnormal fluctuations. The emotional feedback fluctuation index is calculated. The expression is: ; In the formula, It is the sentiment feedback fluctuation index.

8. The material recognition method based on a multimodal large model according to claim 7 is characterized in that: The conversion correlation anomaly index and the emotional feedback fluctuation index are normalized so that they are both between [0,1]. The audit effect analysis factor of the material is calculated based on the normalized conversion correlation anomaly index and the emotional feedback fluctuation index.

9. The material recognition method based on a multimodal large model according to claim 8, characterized in that: Compare the audit effect analysis factor of the acquired material with the audit effect analysis factor reference value preset based on historical data. If the audit effect analysis factor of the material is greater than or equal to the preset audit effect analysis factor reference value, it means that the audit effect of the material is excellent; if the audit effect analysis factor of the material is less than the preset audit effect analysis factor reference value, it means that the audit effect of the material is poor.

Images