Advertisement material identification method and device, equipment and storage medium
By extracting multimodal features from video advertising materials and fusing them with a hierarchical attention mechanism, the problems of misjudgment and missed judgment in advertising material recognition in existing technologies have been solved, achieving more accurate automated recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 广州三七极耀网络科技有限公司
- Filing Date
- 2025-12-31
- Publication Date
- 2026-05-19
AI Technical Summary
Existing technologies struggle to accurately and comprehensively detect AI-generated elements in advertising materials, leading to frequent misjudgments and omissions, and failing to meet increasingly stringent compliance requirements.
By extracting features from multiple modalities of video advertising materials, including visual, textual, audio, and spatiotemporal features, and combining them with a hierarchical attention mechanism to perform feature fusion, fused features are generated and identified, ultimately determining whether the material is artificial intelligence-generated material.
It significantly improves the accuracy of advertising creative identification, reduces the probability of false positives and false negatives, and achieves more comprehensive automated analysis.
Smart Images
Figure CN122066469A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data recognition technology, and in particular to a method, apparatus, device and storage medium for recognizing advertising materials. Background Technology
[0002] With the rapid development of artificial intelligence (AI) technology, AI-generated content is being used more and more widely in advertising creative production. However, it has also brought a series of related issues, especially regarding the compliance of advertising platforms. Generally, regulatory agencies place higher demands on the transparency and traceability of AI-generated content, as unlabeled AI-generated content may damage brand image and consumer trust.
[0003] In related technologies, advertising material review systems mainly rely on manual review or simple image / text recognition technology, which makes it difficult to accurately and comprehensively detect AI-generated elements in the materials, easily leading to false positives and false negatives. This increases the difficulty of supervision, poses potential risks to advertisers and consumers, and fails to meet increasingly stringent compliance requirements, necessitating improvement. Summary of the Invention
[0004] This application provides an advertising material identification method, apparatus, device, and storage medium, which solves the technical problem in related technologies that it is difficult to accurately and comprehensively detect AI-generated elements in the material, and that misjudgment and missed judgment are easy to occur. By extracting features from multiple different modalities of the uploaded video advertising material, the method determines whether the advertising material is an artificial intelligence synthesized material based on the recognition results of single modal features and the recognition results of fused features, thereby improving the accuracy of material identification and significantly reducing the probability of misjudgment and missed judgment.
[0005] Firstly, this application provides a method for identifying advertising creative materials, including: The uploaded video advertising material is obtained, and features of multiple different modalities are extracted from the video advertising material to obtain multiple single modal features. Each single modal feature is identified to obtain a first identification result. Based on a hierarchical attention mechanism, the multiple single-modal features are fused to obtain fused features, and the fused features are then identified to obtain a second identification result; Based on the first identification result and the second identification result, it is determined whether the advertising material is an artificial intelligence-generated material.
[0006] Optionally, the step of extracting features from the video advertising material using multiple different modalities to obtain multiple single-modal features includes at least two or more of the following: Visual features are extracted from the images in the video advertising material to obtain visual features; Text features are extracted from the text in the video advertisement material to obtain semantic features; Audio features are extracted from the audio in the video advertisement material to obtain audio features; Spatiotemporal features are extracted from the video in the video advertising material to obtain spatiotemporal features.
[0007] Optionally, the step of identifying each of the single modal features to obtain a first identification result includes: In the case where the image includes a human face, the visual features are identified to determine whether it is a synthetic human face; The semantic features are identified to determine whether they meet the conditions for marketing copywriting. The audio features are identified to determine whether intelligent synthesized speech exists; The spatiotemporal features are identified to determine whether intelligent synthetic animation exists.
[0008] Optionally, the step of extracting features from the video advertising material using multiple different modalities to obtain multiple single-modal features includes: The video advertising material is divided into multiple video segments, and multiple single-modal features are obtained by extracting features from the video segments in multiple different modalities. Accordingly, the feature fusion of the multiple single-modal features based on the hierarchical attention mechanism to obtain fused features includes: Determine the temporal weight vector corresponding to each single modal feature contained in the video segment; Based on the temporal weight vector, key time points are determined, and tensor fusion is performed on the single modal features corresponding to the key time points to generate key time features.
[0009] Optionally, determining the temporal weight vector corresponding to each single modal feature contained in the video segment includes: Calculate the dominant value of each single modal feature in the video segment; The temporal weight vector corresponding to each single modal feature is generated based on the dominant value of each single modal feature.
[0010] Optionally, the step of determining key time points based on the temporal weight vector and performing tensor fusion on the single modal features corresponding to the key time points to generate key time features includes: The moment point in the temporal weight vector where the weight change value is greater than a preset threshold is determined as the key moment point, and a unified three-dimensional tensor is constructed for each single modal feature corresponding to the key moment point. The three-dimensional tensor is adjusted into a one-dimensional long vector, and the one-dimensional long vector is subjected to dimensionality reduction and nonlinear transformation to obtain key moment features.
[0011] Optionally, the first identification result is dictionary-structured data containing detection results corresponding to each single modal feature, and the second identification result is probability distribution data. The step of determining whether the advertising material is an AI-generated material based on the first and second identification results includes: The second recognition result is vectorized to determine the difference score between the first recognition result and the vectorized second recognition result; If the difference score is less than the preset score, the first identification result and the vectorized second identification result are concatenated, and the concatenation result is input into the adjudication network model to obtain the judgment result of whether it is artificial intelligence synthetic material.
[0012] Secondly, this application provides an advertising material recognition device, comprising: The acquisition module is used to acquire uploaded video advertising materials; The first recognition module is used to extract features from the video advertising material in multiple different modalities to obtain multiple single modal features, and to recognize each single modal feature to obtain a first recognition result; The second recognition module is used to fuse the multiple single-modal features based on a hierarchical attention mechanism to obtain fused features, and to recognize the fused features to obtain a second recognition result. The material determination module is used to determine whether the advertising material is an artificial intelligence-generated material based on the first identification result and the second identification result.
[0013] Thirdly, this application also provides an advertising material recognition device, the device comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the advertising material recognition method as described in any of the preceding first aspects.
[0014] Fourthly, this application also provides a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the advertising material identification method as described in any of the first aspects above.
[0015] In the solution provided in this application embodiment, for uploaded video advertising materials, multiple single-modal features are first extracted from multiple different modalities to obtain multiple single-modal features. Each single-modal feature is identified to obtain a first identification result. Then, based on a hierarchical attention mechanism, the multiple single-modal features are fused to obtain fused features. The fused features are identified to obtain a second identification result. Finally, based on the first identification result and the second identification result, it is determined whether the advertising material is artificial intelligence-generated material. In the process of advertising material identification, a multimodal mechanism is introduced, which simultaneously combines the identification results of multiple single-modal features and the identification results of multimodal fused features to jointly identify the advertising material, so as to ultimately determine whether it is artificial intelligence-generated material. This achieves a more comprehensive analysis of advertising materials, has a higher degree of automation, improves the accuracy of material identification, and significantly reduces the probability of false positives and false negatives. Attached Figure Description
[0016] Figure 1 This is a flowchart of an advertising material identification method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a method for determining and recognizing multiple single-modal features according to an embodiment of this application; Figure 3 This is a flowchart of another advertising material identification method provided in the embodiments of this application; Figure 4 This is a flowchart illustrating a method for determining whether advertising material is artificial intelligence-generated material, as provided in an embodiment of this application. Figure 5 This is a block diagram of the module structure of an advertising material recognition device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an advertising material recognition device provided in an embodiment of this application. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as being processed sequentially, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. A process can be terminated when its operation is completed, but it may also have additional steps not included in the drawings. A process can correspond to a method, function, procedure, subroutine, subroutine, etc.
[0018] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0019] The advertising material identification method provided in this application can be applied to scenarios requiring advertising material identification, such as information publishing platforms identifying materials uploaded by various content creators to determine whether they are AI-generated. Therefore, efficiently detecting and identifying uploaded video advertising materials to determine their compliance has become a pressing issue for various service providers. With the increasing complexity of content creation and the evolution of AI capabilities, traditional methods struggle to accurately identify complex AI-generated content, easily leading to misjudgments and omissions, resulting in the release of non-compliant advertising materials and the risk of regulatory penalties. Manual verification is time-consuming and prone to errors when dealing with large volumes of materials, impacting overall work efficiency. Existing single-modal analysis cannot comprehensively assess the authenticity of materials; for example, analyzing only images may overlook AI-generated traces in text, failing to meet regulatory requirements for comprehensive transparency of AI content.
[0020] The advertising material identification method provided in this application embodiment can be executed by a device with computing power, such as a server, desktop computer, or laptop computer.
[0021] Figure 1This is a flowchart illustrating an advertising material identification method provided in an embodiment of this application. (Reference) Figure 1 The advertising material identification method includes: Step S101: Obtain the uploaded video advertising material, extract features from the video advertising material in multiple different modalities to obtain multiple single modal features, and identify each single modal feature to obtain the first identification result.
[0022] The video ad creative can be a user-uploaded video file to be identified and analyzed. This video file contains multimodal information, such as visual images, audio, and text (subtitles / on-screen text). The identification of this video ad creative yields corresponding results, such as whether it was generated by AI.
[0023] Feature extraction involves extracting meaningful, machine-understandable structured information, such as feature vectors, from the original video ad creative. In this step, it involves extracting features from multiple modalities of the video ad creative to obtain multiple single-modal features. A single-modal feature refers to feature data extracted from a specific modality of the video ad creative (e.g., visual only, audio only). Examples include feature vectors representing visual content and feature vectors representing audio waveforms.
[0024] Optionally, modal settings can be pre-defined, meaning multiple modalities for feature extraction can be set. When data of the corresponding modality exists in the video ad material, the corresponding feature is extracted to obtain the corresponding single-modal feature. Optionally, modalities such as image, text, audio, and video can be set. Different modalities have corresponding pre-defined feature extraction methods; using the appropriate feature extraction method will extract the corresponding single-modal feature.
[0025] In this process, after obtaining multiple single-modal features, each single-modal feature is identified to obtain a first identification result. This first identification result is then used to combine with the second identification result corresponding to the multimodal fusion feature to jointly determine the final advertising material identification result. For each single-modal feature, a corresponding identification method is pre-set, and the identification of the corresponding single-modal feature based on the corresponding identification method can obtain the corresponding identification result.
[0026] Step S102: Based on the hierarchical attention mechanism, multiple single-modal features are fused to obtain fused features, and the fused features are identified to obtain a second identification result.
[0027] In one embodiment, multiple single-modal features are obtained and fused using a hierarchical attention mechanism to obtain a fused feature. The fused feature is a unified and comprehensive new feature representation that integrates multiple single-modal features, such as visual features, semantic features, audio features, and spatiotemporal features, through a hierarchical attention mechanism.
[0028] Optionally, when determining single-modal features, the following approach can be adopted: divide the video advertising material into multiple video segments, and extract features from each video segment using multiple different modalities to obtain multiple single-modal features. The video segments can be divided according to a set fixed duration, meaning that each of the resulting single-modal features corresponds to one video segment. Correspondingly, the process of fusing these multiple single-modal features using a hierarchical attention mechanism to obtain fused features can be as follows: determine the temporal weight vector corresponding to each single-modal feature contained in the video segment, determine key moment points based on the temporal weight vectors, and perform tensor fusion on the single-modal features corresponding to the key moment points to generate key moment features. These key moment features are the fused features obtained by fusing multiple single-modal features.
[0029] Step S103: Determine whether the advertising material is an artificial intelligence-generated material based on the first recognition result and the second recognition result.
[0030] In one embodiment, after obtaining the first identification result and the second identification result, the first identification result and the second identification result are combined to determine whether the advertising material is AI-generated material. Optionally, the first identification result, for each single modal feature, can be multiple specific result scores, and the second identification result is a unified result score. The multiple result scores are added to the unified result score to obtain the final judgment score. When the judgment score is greater than a preset judgment value, the advertising material is determined to be AI-generated material; otherwise, the advertising material is determined not to be AI-generated material. Optionally, the first identification result and the second identification result can also be specific results determining whether AI-generated content exists. When either result is determined to contain AI content, the corresponding final judgment result is that the advertising material is determined to be AI-generated material. That is, in addition to obtaining a judgment result by conventionally identifying single modal features, further identification of fused features is performed to obtain a judgment result, jointly determining the final result to prevent missed judgments.
[0031] As described above, for uploaded video advertising materials, multiple single-modal features are first extracted from different modalities to obtain multiple single-modal features. Each single-modal feature is then identified to obtain a first identification result. Next, based on a hierarchical attention mechanism, the multiple single-modal features are fused to obtain a fused feature. The fused feature is then identified to obtain a second identification result. Finally, based on the first and second identification results, it is determined whether the advertising material is AI-generated material. In the process of advertising material identification, a multimodal mechanism is introduced, which simultaneously combines the identification results of multiple single-modal features and the identification results of multimodal fused features to jointly identify the advertising material, so as to ultimately determine whether it is AI-generated material. This achieves a more comprehensive analysis of advertising materials, has a higher degree of automation, improves the accuracy of material identification, and significantly reduces the probability of false positives and false negatives.
[0032] Figure 2 This is a flowchart illustrating a method for determining and recognizing multiple single-modal features according to an embodiment of this application, such as... Figure 2 As shown, it includes: Step S201: Obtain the uploaded video advertising material, extract visual features from the images in the video advertising material to obtain visual features, extract text features from the text in the video advertising material to obtain semantic features, extract audio features from the audio in the video advertising material to obtain audio features, and extract spatiotemporal features from the video in the video advertising material to obtain spatiotemporal features.
[0033] Among them, visual features can be extracted features such as texture, edge, and color; semantic features can be extracted features such as keywords, themes, and emotions; audio features can be extracted features such as pitch, timbre, and rhythm; and spatiotemporal features can be extracted features such as actions and scene changes.
[0034] Optionally, visual feature extraction can be achieved using deep learning-based feature extraction methods that automatically extract visual features from video footage using neural networks. Text feature extraction can be implemented using methods such as bag-of-words model, TF-IDF, N-gram model, and Word2Vec word embedding. Audio feature extraction can be achieved using temporal feature extraction, frequency domain feature extraction, or cepstral analysis. Spatiotemporal feature extraction can be achieved using a 3D convolutional neural network, where the input video clip is slid across the time, height, and width dimensions to capture both spatial appearance and short-term motion information to obtain spatiotemporal features.
[0035] Step S202: If the image includes a face, identify the visual features to determine whether it is a synthetic face, identify the semantic features to determine whether it meets the conditions of marketing copywriting, identify the audio features to determine whether there is intelligent synthetic speech, and identify the spatiotemporal features to determine whether there is intelligent synthetic animation.
[0036] In the visual feature recognition process, when the image includes a face, a pre-trained deep forgery detection model (such as MesoNet or an EfficientNet-based detector) can be used to analyze the features of the extracted face region. By capturing unnatural textures, boundary artifacts, or inconsistencies in physiological signals, it can be determined whether the face is synthetic. In the semantic feature recognition process, the text extracted by OCR and the speech text transcribed by ASR are input into an NLP model. Keyword matching or text classification models are used to determine whether the conditions of marketing copywriting are met (such as containing prohibited words or exaggerated claims). In the audio feature recognition process, audio deep learning models (such as the solution in the ASVspoof challenge) can be used to analyze features such as the Mel spectrum to detect whether there are typical traces of intelligent synthesized speech (such as discontinuous spectrum or lack of natural rhythm). In the spatiotemporal feature recognition process, 3D CNN or temporal models can be used to analyze the motion flow of continuous frames to detect whether the movement of objects or the actions of people violate physical laws or have temporal inconsistencies, thereby determining whether there is intelligent synthesized animation.
[0037] Step S203: Based on the hierarchical attention mechanism, multiple single-modal features are fused to obtain fused features, and the fused features are identified to obtain a second identification result.
[0038] Step S204: Determine whether the advertising material is an artificial intelligence-generated material based on the first recognition result and the second recognition result.
[0039] As described above, for uploaded video ad creatives, visual features are extracted from the images to obtain visual features; text features are extracted from the text to obtain semantic features; audio features are extracted from the audio to obtain audio features; spatiotemporal features are extracted from the video to obtain spatiotemporal features; when the image includes a face, visual features are identified to determine if it is a synthetic face; semantic features are identified to determine if it meets the conditions for marketing copy; audio features are identified to determine if there is intelligently synthesized speech; and spatiotemporal features are identified to determine if there is intelligently synthesized animation. Then, based on layering... The attention mechanism fuses multiple single-modal features to obtain fused features, identifies the fused features to obtain a second identification result, and finally determines whether the advertising material is AI-generated material based on whether it contains a synthetic face, whether it is marketing copy, whether it exists, and intelligent synthetic animation, together with the second identification result. In the process of advertising material identification, a multimodal mechanism is introduced to simultaneously combine the identification results of multiple single-modal features and the identification results of multimodal fused features to jointly identify advertising materials, so as to ultimately determine whether it is AI-generated material. This achieves a more comprehensive analysis of advertising materials, has a higher degree of automation, improves the accuracy of material identification, and significantly reduces the probability of false positives and false negatives.
[0040] Figure 3 This is a flowchart of another advertising material identification method provided in the embodiments of this application, such as... Figure 3 As shown, it includes: Step S301: Obtain the uploaded video advertising material, divide the video advertising material into multiple video segments, extract features of multiple different modalities from the video segments to obtain multiple single modal features, and identify each single modal feature to obtain a first identification result.
[0041] Step S302: Calculate the dominant value of each single modal feature in the video segment, and generate the temporal weight vector corresponding to each single modal feature based on the dominant value of each single modal feature.
[0042] For example, taking a video ad creative of length T seconds as an example, the video ad creative is evenly divided into N time segments, for example, one segment per second, or one segment every 0.5 seconds. That is, assuming the total number of segments is N, for the nth segment (corresponding to time t_n), feature extraction is performed simultaneously to obtain feature vectors of three modalities: visual feature V_n ∈ R^d_v, audio feature A_n ∈ R^d_a, and semantic feature T_n ∈ R^d_t. For the total number of segments, three feature sequences are obtained: {V_1, V_2,..., V_N}, {A_1, A_2, ..., A_N}, and {T_1, T_2, ..., T_N}.
[0043] The dominance value reflects whether a modality is "dominant" at a given time, depending on its own information strength and its degree of difference from other modalities. For each modal feature at each time t_n, its dominance value is calculated as: D_n^m (m ∈ {v, a, t}). The dominance value D_n^m can be calculated as: D_n^m = α * S_abnormal(F_n^m) + β * S_divergence(F_n^m). Here, F_n^m represents the corresponding modal feature, α and β are balancing hyperparameters (e.g., α = 0.7, β = 0.3), S_abnormal(F_n^m) represents the calculated self-abnormality score, and S_divergence(F_n^m) represents the calculated cross-modal dissimilarity score. S_abnormal(F_n^m) measures whether the feature is AI-synthesized; S_divergence(F_n^m) measures the inconsistency between the feature and the other two modal features.
[0044] Optionally, when calculating the anomalousness score, each segment feature can be scored using a pre-trained single-modal AI generation detector. Taking an image as an example, S_abnormal(V_n) represents the probability that the image segment is judged as AI-generated. The higher the score, the greater the probability of being judged as AI-generated, and the stronger the dominance. Optionally, taking the aforementioned three modal features as an example, the cross-modal dissimilarity score calculation process can be to calculate the cosine distance or Euclidean distance between the current modal feature and the other two modal features after projection into the common subspace, and take the average of the calculation results as the cross-modal dissimilarity score of the current modal feature. Through the above calculation, for a time t_n, its corresponding dominance value is denoted as [D_n^v, D_n^a, D_n^t]. The dominant value is then normalized, for example, by performing Softmax processing on the three dominant values, to obtain the temporal weight vector at that time step, denoted as W_n=[w_n^v, w_n^a, w_n^t], where w_n^v + w_n^a + w_n^t = 1. After calculation at each time step, the corresponding weight sequence {W_1, W_2, ..., W_N} can be output, with each W_n being a three-dimensional vector.
[0045] Step S303: Determine the time points in the time series weight vector where the weight change value is greater than the preset threshold as key time points. Construct a unified three-dimensional tensor for each single modal feature corresponding to the key time point. Adjust the three-dimensional tensor into a one-dimensional long vector. Perform dimensionality reduction and nonlinear transformation on the one-dimensional long vector to obtain key time features. Identify the key time features to obtain the second identification result.
[0046] In one embodiment, for each mode m, the absolute difference of its weight at consecutive time points is calculated as the weight change value, i.e., Δw_n^m = |w_{n+1}^m - w_n^m|, and the preset threshold is θ, for example, 0.3. For any time t_n, if the weight change value max(Δw_n^v, Δw_n^a, Δw_n^t) of any mode is greater than θ, then time t_n is marked as a critical time point.
[0047] For a single key moment t_k, to preserve their respective feature information, each feature vector is supplemented with a dimension of 1, i.e.: V_k' = [V_k, 1] ∈ R^(d_v+1), A_k' = [A_k, 1] ∈ R^(d_a+1), T_k' = [T_k, 1] ∈ R^(d_t+1). The outer product of the three is calculated to generate a three-dimensional fusion tensor Z_k, which explicitly encodes the product interaction relationships between all dimensions of the three modalities. The three-dimensional tensor Z_k is then reshaped into a one-dimensional long vector z_flat_k, where: z_flat_k = flatten(Z_k) ∈ R^((d_v+1)*(d_a+1)*(d_t+1)). The calculated result z_flat_k is then input into a small multilayer perceptron for dimensionality reduction and nonlinear transformation, i.e.: h_k = MLP_fusion(z_flat_k). The output of the perceptron, h_k ∈ R^256, is the fused deep feature of the key moment t_k.
[0048] Accordingly, after obtaining the key moment feature, each key moment feature h_k can be input into a set key point classifier, such as a binary classifier, and the output result is the recognition result generated by AI or created by the user.
[0049] Step S304: Determine whether the advertising material is an artificial intelligence-generated material based on the first recognition result and the second recognition result.
[0050] As described above, for uploaded video advertising materials, multiple modal features are first extracted to obtain multiple single-modal features. Each single-modal feature is then identified to obtain a first recognition result. Next, the dominant value of each single-modal feature in the video segment is calculated. Based on the dominant value of each single-modal feature, a temporal weight vector corresponding to each single-modal feature is generated. The time points where the weight change value in the temporal weight vector exceeds a preset threshold are determined as key time points. A unified three-dimensional tensor is constructed for each single-modal feature corresponding to the key time points. The three-dimensional tensor is then adjusted into a one-dimensional long vector, and the one-dimensional long vector is further processed... Dimensionality reduction and nonlinear transformation are used to obtain key moment features. These key moment features are then identified to obtain a second identification result. Finally, based on the first and second identification results, it is determined whether the advertising material is AI-generated material. In the process of advertising material identification, a multimodal mechanism is introduced, which simultaneously combines the identification results of multiple single-modal features and the identification results of multimodal fusion features to jointly identify the advertising material and ultimately determine whether it is AI-generated material. This achieves a more comprehensive analysis of advertising materials, a higher degree of automation, improves the accuracy of material identification, and significantly reduces the probability of false positives and false negatives.
[0051] Figure 4This is a flowchart illustrating a method for determining whether advertising material is artificial intelligence-generated material, as provided in an embodiment of this application. Figure 4 As shown, it includes: Step S401: Obtain the uploaded video advertising material, extract features from the video advertising material in multiple different modalities to obtain multiple single modal features, and identify each single modal feature to obtain the first identification result.
[0052] Step S402: Based on the hierarchical attention mechanism, multiple single-modal features are fused to obtain fused features, and the fused features are identified to obtain a second identification result.
[0053] Step S403: The second recognition result is vectorized to determine the difference score between the first recognition result and the vectorized second recognition result. If the difference score is less than the preset score, the first recognition result and the vectorized second recognition result are concatenated. The concatenated result is input into the adjudication network model to obtain the judgment result of whether it is artificial intelligence synthesized material.
[0054] In one embodiment, the first recognition result is dictionary structure data containing the detection results corresponding to each single modality feature, and the second recognition result is probability distribution data.
[0055] In one embodiment, the second recognition result is vectorized. Specifically, the second recognition result, which includes the multimodal judgment AI generation probability, is expanded into an evidence vector with fixed dimensions. Each dimension corresponds to a specific modality of AI-generated content type, and its value is filled by the detection confidence of the modality feature of that type. Optionally, the absolute difference between the comprehensive AI generation probability of the first recognition result and the weighted average of the confidence of all dimensions in the vectorized second recognition result can be calculated as the difference score. If the difference score is less than a preset score, it indicates that the judgments of the two recognition paths are basically consistent. In this case, the probability distribution vector of the first recognition result and the evidence vector of the second recognition result are concatenated to form a joint feature vector that integrates global comprehensive judgment and local fine-grained evidence. Further, this joint feature vector is input into a lightweight adjudication network model, such as a multilayer perceptron with several hidden layers. After training, this model can directly output the final binary classification judgment result and its confidence score of whether the advertising material is AI-synthesized material based on the deep pattern of the concatenated vector.
[0056] As described above, for uploaded video advertising materials, multiple single-modal features are first extracted from different modalities to obtain multiple single-modal features. Each single-modal feature is then identified to obtain a first identification result. Next, based on a hierarchical attention mechanism, the multiple single-modal features are fused to obtain a fused feature. This fused feature is then identified to obtain a second identification result. The second identification result is then vectorized, and the difference score between the first identification result and the vectorized second identification result is determined. If the difference score is less than a preset score, the first identification result and the vectorized second identification result are concatenated. The concatenated result is input into a decision network model to determine whether the material is AI-generated. In the process of advertising material identification, a multimodal mechanism is introduced, simultaneously combining the identification results of multiple single-modal features and the identification results of the multimodal fused features to jointly identify the advertising material, ultimately determining whether it is AI-generated. This achieves a more comprehensive analysis of the advertising material, a higher degree of automation, improves the accuracy of material identification, and significantly reduces the probability of false positives and false negatives.
[0057] Figure 5 This is a block diagram of the module structure of an advertising material recognition device provided in an embodiment of this application. The device is used to execute an advertising material recognition method provided in the above embodiment, and has corresponding functional modules and beneficial effects for executing the method. Figure 5 As shown, the device specifically includes: Module 101 is used to retrieve uploaded video advertising materials; The first recognition module 102 is used to extract features from the video advertising material in multiple different modalities to obtain multiple single modal features, and to recognize each single modal feature to obtain a first recognition result; The second recognition module 103 is used to fuse the multiple single-modal features based on a hierarchical attention mechanism to obtain fused features, and to recognize the fused features to obtain a second recognition result. The material determination module 104 is used to determine whether the advertising material is an artificial intelligence-generated material based on the first identification result and the second identification result.
[0058] As described above, for uploaded video advertising materials, multiple single-modal features are first extracted from different modalities to obtain multiple single-modal features. Each single-modal feature is then identified to obtain a first identification result. Next, based on a hierarchical attention mechanism, the multiple single-modal features are fused to obtain a fused feature. The fused feature is then identified to obtain a second identification result. Finally, based on the first and second identification results, it is determined whether the advertising material is AI-generated material. In the process of advertising material identification, a multimodal mechanism is introduced, which simultaneously combines the identification results of multiple single-modal features and the identification results of multimodal fused features to jointly identify the advertising material, so as to ultimately determine whether it is AI-generated material. This achieves a more comprehensive analysis of advertising materials, has a higher degree of automation, improves the accuracy of material identification, and significantly reduces the probability of false positives and false negatives.
[0059] In one possible embodiment, the first identification module 102 is specifically used for: Visual features are extracted from the images in the video advertising material to obtain visual features; Text features are extracted from the text in the video advertisement material to obtain semantic features; Audio features are extracted from the audio in the video advertisement material to obtain audio features; Spatiotemporal features are extracted from the video in the video advertising material to obtain spatiotemporal features.
[0060] In one possible embodiment, the first identification module 102 is specifically used for: In the case where the image includes a human face, the visual features are identified to determine whether it is a synthetic human face; The semantic features are identified to determine whether they meet the conditions for marketing copywriting. The audio features are identified to determine whether intelligent synthesized speech exists; The spatiotemporal features are identified to determine whether intelligent synthetic animation exists.
[0061] In one possible embodiment, the first identification module 102 is specifically used for: The video advertising material is divided into multiple video segments, and multiple single-modal features are obtained by extracting features from the video segments in multiple different modalities. Accordingly, the second identification module 103 is specifically used for: Determine the temporal weight vector corresponding to each single modal feature contained in the video segment; Based on the temporal weight vector, key time points are determined, and tensor fusion is performed on the single modal features corresponding to the key time points to generate key time features.
[0062] In one possible embodiment, the first identification module 102 is specifically used for: Calculate the dominant value of each single modal feature in the video segment; The temporal weight vector corresponding to each single modal feature is generated based on the dominant value of each single modal feature.
[0063] In one possible embodiment, determining key time points based on the temporal weight vector and performing tensor fusion on the single modal features corresponding to the key time points to generate key time features includes: The moment point in the temporal weight vector where the weight change value is greater than a preset threshold is determined as the key moment point, and a unified three-dimensional tensor is constructed for each single modal feature corresponding to the key moment point. The three-dimensional tensor is adjusted into a one-dimensional long vector, and the one-dimensional long vector is subjected to dimensionality reduction and nonlinear transformation to obtain key moment features.
[0064] In one possible embodiment, the first recognition result is dictionary-structured data containing detection results corresponding to each individual modal feature, the second recognition result is probability distribution data, and the material determination module 104 is specifically used for: The second recognition result is vectorized to determine the difference score between the first recognition result and the vectorized second recognition result; If the difference score is less than the preset score, the first identification result and the vectorized second identification result are concatenated, and the concatenation result is input into the adjudication network model to obtain the judgment result of whether it is artificial intelligence synthetic material.
[0065] Figure 6 This is a schematic diagram of the structure of an advertising material recognition device provided in an embodiment of this application, such as... Figure 6 As shown, the device includes a processor 201, a memory 202, an input device 203, and an output device 204; the number of processors 201 in the device can be one or more. Figure 6 Taking a processor 201 as an example; the processor 201, memory 202, input device 203, and output device 204 in the device can be connected via a bus or other means. Figure 6Taking a bus connection as an example, the memory 202, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions or modules corresponding to the advertising material recognition method in this embodiment. The processor 201 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 202, thereby realizing the aforementioned advertising material recognition method. The input device 203 can be used to receive input digital or character information and generate key signal inputs related to user settings and function control of the device. The output device 204 may include a display screen or other display device.
[0066] This application embodiment also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform an advertising material identification method, the method comprising: The uploaded video advertising material is obtained, and features of multiple different modalities are extracted from the video advertising material to obtain multiple single modal features. Each single modal feature is identified to obtain a first identification result. Based on a hierarchical attention mechanism, the multiple single-modal features are fused to obtain fused features, and the fused features are then identified to obtain a second identification result; Based on the first identification result and the second identification result, it is determined whether the advertising material is an artificial intelligence-generated material.
[0067] It is worth noting that in the above-described embodiment of an advertising material identification method system, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this application.
[0068] Note that the above are merely preferred embodiments and the technical principles applied in this application. Those skilled in the art will understand that the embodiments of this application are not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the embodiments of this application. Therefore, although the embodiments of this application have been described in detail through the above embodiments, the embodiments of this application are not limited to the above embodiments. More other equivalent embodiments may be included without departing from the concept of the embodiments of this application, and the scope of the embodiments of this application is determined by the scope of the appended claims.
Claims
1. A method for identifying advertising materials, characterized in that, include: The uploaded video advertising material is obtained, and features of multiple different modalities are extracted from the video advertising material to obtain multiple single modal features. Each single modal feature is identified to obtain a first identification result. Based on a hierarchical attention mechanism, the multiple single-modal features are fused to obtain fused features, and the fused features are then identified to obtain a second identification result; Based on the first identification result and the second identification result, it is determined whether the advertising material is an artificial intelligence-generated material.
2. The advertising material identification method according to claim 1, characterized in that, The process of extracting features from the video advertising material using multiple different modalities yields multiple single-modal features, including at least two or more of the following: Visual features are extracted from the images in the video advertisement material to obtain visual features; Text features are extracted from the text in the video advertisement material to obtain semantic features; Audio features are extracted from the audio in the video advertisement material to obtain audio features; Spatiotemporal features are extracted from the video in the video advertising material to obtain spatiotemporal features.
3. The advertising material identification method according to claim 2, characterized in that, The step of identifying each of the single modal features to obtain a first identification result includes: In the case where the image includes a human face, the visual features are identified to determine whether it is a synthetic human face; The semantic features are identified to determine whether they meet the conditions for marketing copywriting. The audio features are identified to determine whether intelligent synthesized speech exists; The spatiotemporal features are identified to determine whether intelligent synthetic animation exists.
4. The advertising material identification method according to any one of claims 1-3, characterized in that, The process of extracting features from the video advertising material using multiple different modalities to obtain multiple single-modal features includes: The video advertising material is divided into multiple video segments, and multiple single-modal features are obtained by extracting features from the video segments in multiple different modalities. Accordingly, the feature fusion of the multiple single-modal features based on the hierarchical attention mechanism to obtain fused features includes: Determine the temporal weight vector corresponding to each single modal feature contained in the video segment; Based on the temporal weight vector, key time points are determined, and tensor fusion is performed on the single modal features corresponding to the key time points to generate key time features.
5. The advertising material identification method according to claim 4, characterized in that, Determining the temporal weight vector corresponding to each single modal feature contained in the video segment includes: Calculate the dominant value of each single modal feature in the video segment; The temporal weight vector corresponding to each single modal feature is generated based on the dominant value of each single modal feature.
6. The advertising material identification method according to claim 4, characterized in that, The step of determining key time points based on the temporal weight vector and generating key time features by tensor fusion of the single modal features corresponding to the key time points includes: The moment point in the temporal weight vector where the weight change value is greater than a preset threshold is determined as the key moment point, and a unified three-dimensional tensor is constructed for each single modal feature corresponding to the key moment point. The three-dimensional tensor is adjusted into a one-dimensional long vector, and the one-dimensional long vector is subjected to dimensionality reduction and nonlinear transformation to obtain key moment features.
7. The advertising material identification method according to any one of claims 1-3, characterized in that, The first identification result is dictionary-structured data containing detection results corresponding to each single modal feature, and the second identification result is probability distribution data. The step of determining whether the advertising material is AI-generated material based on the first and second identification results includes: The second recognition result is vectorized to determine the difference score between the first recognition result and the vectorized second recognition result; If the difference score is less than the preset score, the first identification result and the vectorized second identification result are concatenated, and the concatenation result is input into the adjudication network model to obtain the judgment result of whether it is artificial intelligence synthetic material.
8. An advertising material recognition device, applied to a client-side application, characterized in that, include: The acquisition module is used to acquire uploaded video advertising materials; The first recognition module is used to extract features from the video advertising material in multiple different modalities to obtain multiple single modal features, and to recognize each single modal feature to obtain a first recognition result; The second recognition module is used to fuse the multiple single-modal features based on a hierarchical attention mechanism to obtain fused features, and to recognize the fused features to obtain a second recognition result. The material determination module is used to determine whether the advertising material is an artificial intelligence-generated material based on the first identification result and the second identification result.
9. An advertising material recognition device, the device comprising: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the advertising material identification method as described in any one of claims 1-7.
10. A storage medium storing computer-executable instructions, which, when executed by a computer processor, are used to perform the advertising material identification method as described in any one of claims 1-7.