Advertisement video badness-oriented fine-grained detection method based on knowledge graph and large language model

Abstract

The invention relates to the field of digital content security and artificial intelligence, and discloses an advertisement video badness-oriented fine-grained detection method based on a knowledge graph and a large language model, and the method comprises the steps: 1, collecting an advertisement video, and obtaining a knowledge graph of a badness-oriented advertisement video in a triple form through processing; 2, carrying out semantic segmentation on an advertisement video to be detected; step 3, extracting and processing audio emotion, vision and text information, and retrieving an associated violation mode sub-graph from the knowledge graph according to the advertisement type of the video to be detected; 4, the extracted information features and the knowledge graph sub-graphs in the step 3 are input into a large model, the large model outputs a bad video fine-grained analysis result, and a detection report is generated. According to the method, a specific bad guiding fragment is positioned to support fine-grained traceability, an evidence chain for omnibearing bad guiding detection is generated, and the detection accuracy is improved. The automatic detection of the bad guidance illegal advertisement is realized, the bad guidance illegal clues in the advertisement are accurately positioned, and the manual interpretation is assisted.

Description

Technical Field

[0001] The present invention relates to the fields of digital content security and artificial intelligence, and specifically to a fine-grained detection method for inappropriate content in advertising videos based on knowledge graphs and large language models. Background Art

[0002] Existing advertising review technologies mostly focus on analyzing explicit violation clues in advertising videos, such as filtering out violation texts based on keyword matching rule engines. Unlike traditional illegal and non-compliant advertisements, the violations of bad-oriented advertisements are implicitly expressed, mostly with suggestive clues, and cannot be directly interpreted using legal provisions such as the Advertising Law. In actual supervision work, it mainly relies on manual screening by advertising law enforcement personnel. When a new advertising video is input, it is impossible to quickly match the typical violation pattern of its type, nor can it convert the violation elements of similar historical advertisements into inferable auxiliary clues. It is time-consuming, labor-intensive, and inefficient. For this reason, this application proposes a fine-grained detection method for bad-oriented advertising videos based on knowledge graphs and large language models. Summary of the Invention

[0003] The purpose of the present invention is to provide a fine-grained detection method for bad advertising video content based on knowledge graphs and large language models to solve the problems raised in the above-mentioned background technology.

[0004] To achieve the above objectives, the present invention provides the following technical solutions: a fine-grained detection method for bad content in advertising videos based on knowledge graphs and large language models, comprising the following steps: 1. Collection of advertising videos, collecting labeled bad content from a dataset of illegal video advertisements publicly available from regulatory authorities, obtaining multimodal data including video streams, audio streams, subtitles, and regulatory comments, and processing the data to obtain a knowledge graph of bad content in the form of triples; 2. Video semantic segmentation, performing two-stage video segmentation on videos longer than 30 seconds: in the first stage, coarse-grained segmentation of the video based on shot switching to obtain n video segments [Video1, Video2, ..., Video3]. n ], the second step is to fine-grainedly merge semantically similar video segments to obtain semantically complete video segments [Video 1’ ,Video 2’ ,...,Video n’ ], where n'≤n; Step 3, extraction and processing of audio emotion, visual and text information; Step 4, inputting the extracted information features and the knowledge graph of Step 1 into the big model, and the big model outputs the fine-grained analysis results of the bad video and generates a detection report.

[0005] As a preferred embodiment: in step 1, the speech in the video is subjected to end-to-end automatic speech recognition technology to obtain text content, the video frame is subjected to a visual language model to generate a natural language description, the combined text is cleaned, and meaningless interjections such as "um, uh" are filtered out based on regular expression matching, and the text is input into the big model together with the video title text and regulatory commentary text. The big model combines the predefined knowledge graph entities, relationships and attributes to extract knowledge from the entities, relationships and attributes in the text, and obtains the entity set E = {e1, e2, ..., e p} and the relation set R={(e i ,r ij ,e j )}, forming a knowledge graph S of bad advertising videos in the form of triples = { <e1,r1,e 1’> , <e2,r2,e 2’> ,..., <e n ,r n ,e n’ >}.

[0006] Preferably, in step 2, the advertising video whose length is less than 30 seconds is not segmented.

[0007] As a preferred embodiment, the audio processing of each video clip after segmentation in step 3 is divided into three stages: audio processing → feature extraction → intonation classification, separating the human voice and background music in the advertisement, and extracting acoustic features H = {h1, h2, ..., h n’}, identify the intonation types through deep neural networks, obtain their probability distribution P(y|H), and select the intonation type V with the highest probability as the final judgment result.

[0008] As a preferred approach, step 3 may indicate that scenes or objects may have undesirable guiding implications at the visual level. A cascaded model of object detection and scene understanding is used to generate spatially annotated visual semantic pairs. Intermediate frames are obtained from each segmented video clip. The model detects products, people, and scene elements O, extracts image frames of these elements, and overlays them onto the original video frame I. A visual language model is then used to generate a textual description D = VLM(I, O) of the image.

[0009] As a preferred method: the text analysis in step 3 uses an open source speech-to-text model to extract the text content in the advertising audio, and based on the modifiers and comparative words with absolutist tendencies as clues to the bad orientation of the advertisement, a predefined absolutist vocabulary list and a comparative vocabulary list are pre-defined to screen out clue words such as "thoroughly, radically, compared to" as text features, which is expressed as T = {t1, t2, ..., t m}, m is the number of clue words.

[0010] As a preference: Step 4 aligns the visual, textual, and auditory feature elements of Step 3 with timestamps, and each video clip Video i All contain the characteristics {V i , D i , T i}, and compare it with the knowledge graph related sub- Figure 1 Input into the large language model.

[0011] As a preferred embodiment: the process of obtaining the relevant subgraph of the knowledge graph is: constructing a retrieval vector q = Embedding (advertising type) based on the label of the advertising type to be tested, calculating its semantic similarity Sim(q,e) = cosine(q,Embedding(e)) with the predefined advertising type entity e∈E in the knowledge graph, and selecting the Top-5 similar entities; secondly, extracting the 1-hop neighborhood subgraph of these entities in the knowledge graph through the graph traversal algorithm to form a triple set SubGraph = {(e,r,e')|e∈Top-5,e'∈N(e)} containing the violation pattern.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] The method of the present invention locates specific bad-direction fragments to support fine-grained traceability, generates an evidence chain for all-round detection of bad-direction, realizes automatic detection of bad-direction illegal and irregular advertisements, accurately locates bad-direction illegal clues in advertisements, and assists manual interpretation. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flow chart of the method of the present invention;

[0015] Figure 2 This is a system framework diagram of the present invention. DETAILED DESCRIPTION

[0016] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0017] Example

[0018] See also Figure 1-Figure 2The method for fine-grained detection of bad-direction advertising videos based on knowledge graph and large language model in the figure includes step 1, collection of advertising videos, collecting annotated bad-direction advertising video data from the illegal video advertising dataset disclosed by the regulatory authorities, obtaining multimodal data including video stream, audio stream, subtitle text and regulatory comments, and processing to obtain a knowledge graph of bad-direction advertising videos in the form of triples; step 2, video semantic segmentation, for videos longer than 30 seconds, two-stage video segmentation is performed: the first stage is to coarsely segment the video based on shot switching to obtain n video segments [Video1, Video2,..., Video n ], the second step is to fine-grainedly merge semantically similar video segments to obtain semantically complete video segments [Video 1’ ,Video 2’ ,...,Video n’ ], where n'≤n; Step 3, extract and process audio emotion, visual and text information, and retrieve the associated violation pattern subgraph from the knowledge graph according to the advertising type of the video to be tested; Step 4, input the extracted information features and the relevant subgraph of the knowledge graph retrieved in Step 3 into the big model, and the big model outputs the fine-grained analysis results of the bad video and generates a detection report.

[0019] In this embodiment, in order to form an evidence chain for detecting potentially negative connotations in advertising videos and provide dynamic and explainable decision support for compliance review of advertising content, it is necessary to extract advertising event units through video semantic segmentation, construct a multimodal feature set including visual scenes, voice emotions, and text semantics, combine domain knowledge graphs, realize collaborative detection of textual hints, visual metaphors, and intonation induction, and complement cross-modal evidence, and combine large language models for causal reasoning. The implementation process is as follows: Figure 1 As shown, it is divided into two core modules:

[0020] The first part is the construction of the knowledge graph: annotated bad-oriented advertising video data is collected from the illegal video advertising dataset disclosed by the regulatory authorities, and multimodal data including video stream, audio stream, subtitle text and regulatory comments are obtained. The text content of the voice in the video is obtained through end-to-end automatic speech recognition technology, and the video frame is used to generate a natural language description through a visual language model. The combined text of the two is cleaned, and meaningless modal particles such as "um, uh" are filtered out based on regular expression matching. It is input into the big model together with the video title text and regulatory comment text. The big model combines the predefined knowledge graph entities, relations and attributes to extract knowledge from the entities, relations and attributes in the text, and obtains the entity set E = {e1, e2, ..., e p} and the relation set R={(e i ,r ij ,e j)}, forming a knowledge graph S of bad-oriented advertising videos in the form of triples = { <e1,r1,e 1’> , <e2,r2,e 2’> ,..., <e n ,r n ,e n’ >}.

[0021] The second part is the multimodal feature extraction of the advertising videos to be tested: the length of the advertising videos to be tested varies, and different segments may have different semantics. For videos longer than 30 seconds, a two-stage video segmentation is performed: the first stage is to coarsely segment the video based on shot switching to obtain n video segments [Video1, Video2, ..., Video n ], the second step is to fine-grainedly merge semantically similar video segments to obtain semantically complete video segments [Video 1’ ,Video 2’ ,...,Video n’ ], where n'≤n. No segmentation is performed on advertising videos shorter than 30 seconds. The videos contain multimodal information such as voice, images, and text. This information reflects possible negative influences from different dimensions and requires comprehensive analysis. The voice and intonation information in advertising videos can often be somewhat provocative to the audience and may serve as an important clue to negative influences. The audio processing of each segmented video clip is divided into three stages: audio processing → feature extraction → intonation classification. The human voice and background music in the advertisement are separated, and the acoustic features H = {h1, h2, …, h n’}, deep neural networks are used to identify intonation types and obtain their probability distribution P(y|H). The intonation type V with the highest probability is selected as the final judgment result. At the visual level, there may be scenes or objects that have negative guidance hints. A cascade model of target detection and scene understanding is used to generate visual semantic pairs with spatial annotations. The intermediate frame is obtained from each segmented video clip. The model detects products, people, and scene elements O, and cuts out the image frames of these elements and superimposes them on the original video frame I. The visual language model is used to generate a text description of the image D = VLM(I,O). For text analysis, an open source speech-to-text model is used to extract text content from advertising audio. Considering that modifiers and comparative words with absolutist tendencies may serve as clues to negative advertising guidance, a predefined absolute vocabulary and comparative vocabulary are used to screen out clue words such as "thoroughly, radically, compared to" as text features, which are expressed as T = {t1, t2, ..., t m}, m is the number of clue words, align the visual, text, and auditory feature elements with timestamps, and each video clip Video i All contain the characteristics {V i , D i, T i}, the process of obtaining the relevant subgraph of the knowledge graph is as follows: construct a retrieval vector q = Embedding (advertising type) based on the ad type label to be tested, calculate its semantic similarity Sim(q,e) = cosine(q,Embedding(e)) with the predefined ad type entity e∈E in the knowledge graph, and select the Top-5 similar entities; secondly, extract the 1-hop neighborhood subgraph of these entities in the knowledge graph through the graph traversal algorithm to form a triple set SubGraph = {(e,r,e')|e∈Top-5,e'∈N(e)} containing the violation pattern, and compare it with the relevant subgraph of the knowledge graph obtained by retrieval. Figure 1 The complete detection method is as follows: Figure 2 shown.

[0022] Furthermore, video data containing bad-oriented advertisements was collected from the illegal video datasets released by regulatory authorities. The Whisper-large model was used to obtain text content for the speech part, and the visual language model MiniCPM-V was used to generate natural language descriptions for the video part. The combined text was cleaned and a regular expression-based matching method was used to directly filter out meaningless interjections such as "hmm, uh". The filtered text was input into the large model together with the video title text and regulatory commentary text. The large model combined the predefined entities, relationships, and attributes to extract knowledge from the entities, relationships, and attributes in the text, and obtained the entity set E = {e1, e2, ..., e p} and the relation set R={(e i ,r ij ,e j )}, forming a knowledge graph S of bad advertising videos in the form of triples = { <e1,r1,e 1’> , <e2,r2,e 2’> ,..., <e n ,r n ,e n’ >}, the predefined entities, relationships, and attributes are as follows:

[0023] Entity types include:

[0024] Ad type (beauty / food / pharmaceuticals, etc.): Attributes include publishing platform and promoter identity;

[0025] Product type (health products / cosmetics / financial products, etc.): Attributes include product name, target group (elderly / young women / young men / children, etc.), and purpose;

[0026] Negative orientation type (feudal superstition / incitement of anxiety / sexism, etc.): Attributes include the degree of negativeness (1-5);

[0027] Relationships include:

[0028] Ad_Contains_Product: The text contains an advertised product;

[0029] Advertisement contains harmful content: The text contains advertising elements and harmful keywords.

[0030] Product_involved_bad_guidance: logical association between product usage and event type;

[0031] Input advertising video. For advertising videos longer than 30 seconds, the video segmentation is divided into two steps: the first step is to use the PySceneDetect library in Python to segment the video based on the shot switching to obtain n video segments [Video1, Video2, ..., Video n ], the second step is to use the ImageBind model to extract each adjacent segment Video i 0.5 second frame before the end and Video i+1 The feature f of the 0.5 second frame at the beginning i and f i+1 , if ||f i -f i+1 ||≤threshold∈[0.8,1.0], the two segments are concatenated into one segment, and the semantically consistent video segments are finally segmented. For advertisement videos with a length of less than 30 seconds, no segmentation is performed. For each segmented video segment, the open source audio separation tool Spleeter is used to separate the human voice and background music in the advertisement. The Wav2Vec2.0 model is used to extract acoustic features such as pitch and spectrum from the separated pure human voice. N}=Wav2Vec2.0(V vocal ), four tones of voice are identified through pre-trained deep neural networks: urgency, intimidation, affinity, and authority, and their probability distributions are obtained P(y|H)=Softmax(W·DNN(H)+b), where W is the weight matrix, b is the bias vector, y∈{urgency, fear, affinity, authority}, and the tone type V with the highest probability is selected as the final judgment result. Three frames are evenly taken from each video clip, and the YOLOv8s model is used to detect products, people, and scene elements. The picture frames of these elements are intercepted and superimposed on the original video frames. The visual language model MiniCPM-V is used to generate the text description of the image D=MiniCPM-V(I,O), where O is the extracted box image of products, people, and scene elements, and I is the original video frame. The Whisper-large model is used to extract the text content Text=Whisper(V vocal), considering that modifiers and comparative words with absolutist tendencies may serve as clues to negative advertising orientation, a predefined absolute vocabulary list and comparative vocabulary list are used to filter out clue words such as "thoroughly, radically, compared to" T = {t1, t2, ..., t m}, m is the number of clue words, align the visual, text, and auditory feature elements with timestamps, and each video clip Video i All contain the characteristics {V i , D i , T i}, where the process of obtaining the relevant subgraph of the knowledge graph is as follows: construct a retrieval vector q = Embedding (advertising type) based on the ad type label to be tested, calculate its semantic similarity Sim(q,e) = cosine(q,Embedding(e)) with the predefined ad type entity e∈E in the knowledge graph, and select the Top-5 similar entities; secondly, extract the 1-hop neighborhood subgraph of these entities in the knowledge graph through the graph traversal algorithm to form a triple set SubGraph = {(e,r,e')|e∈Top-5,e'∈N(e)} containing the violation pattern, and compare it with the relevant subgraph of the knowledge graph obtained by retrieval. Figure 1 The data is input into a large language model to achieve fine-grained detection of inappropriate advertising videos.

[0032] Based on the knowledge graph, we designed a thinking chain reasoning prompt and input it into the large model to detect bad content in advertising videos and generate a detection report. The example is as follows.

[0033] prompt=As a professional ad reviewer, you can identify whether an ad contains negative content. Negative content in ads includes those that create anxiety about appearance, mislead values, create anxiety about education, involve feudal superstition, undermine group unity, hinder social stability, harm the public interest, involve identity discrimination, harm physical and mental health, and promote products prohibited by law. Based on the following ad knowledge graph containing negative content and the characteristics of the ad video, analyze and identify whether the ad video contains negative content and output the search results:

[0034] {Video_Features+Knowledge_Graph}

[0035] Consider:

[0036] 1. Identify potential harmful features in advertising videos and comprehensively scan for the following illegal elements based on timestamps:

[0037] Language features: whether the text contains absolute terms and inductive expressions.

[0038] Acoustic characteristics: whether the tone of voice has abnormal intonation such as fear or authority.

[0039] Visual features: Are there any suggestive image symbols or scenes in the visual description?

[0040] If there are certain illegal elements, please analyze whether this may contain bad guidance in combination with the knowledge graph.

[0041] 2. If it is determined that there is no unhealthy behavior, the reasons are given and the results are directly output. If it is determined that there is unhealthy behavior, please provide a chain of evidence for the violation. The chain of evidence format is [Violation Type Matching] → [Contradiction Analysis] → [Timestamp Association]. Combining multimodal evidence and knowledge graphs, the results are finally presented in a structured form:

[0042] (1) Clear determination of the type of negative guidance and credibility score (1-5)

[0043] (2) Detailed analysis of contradictions

[0044] (3) Positioning of bad guidance periods with accuracy down to the second level.

[0045] Specifically, input: a cosmetics advertising video;

[0046] Description: While promoting the cosmetics product, the ad included a text screen with the message "Fair skin = successful life" and a voiceover saying "Sisters of the same age have already started changing their destiny through whitening."

[0047] Output (test results):

[0048] "Violation Type": ["Spreading Appearance Anxiety"],

[0049] "Spatial Positioning": [{"Start Time":"00:16.3","End Time":"00:19.1",

[0050] "Evidence": ["Matched to historical offending entity <Quick freckle removal, anxious appearance, 4 points>", "Visual appearance of fair skin", "Comparative terms appear", "Urgent tone"]}]).

[0051] According to the categories of beauty, food, medicine, finance, education, games, network services, and tourism, 50 video advertisements were randomly collected in each category. The compliance test was performed using the method proposed in this invention, and manual inspection was used for re-inspection. The specific results are shown in Table 1 below.

[0052] Table 1 Compliance test results of different types of advertisements

[0053]

[0054] As can be seen from Table 1 above, in the compliance detection process of different types of advertisements, the accuracy of the machine inspection results and the accuracy of the manual inspection results are within 5%, and the accuracy rate of the compliance inspection of advertising videos is relatively high. The method of the present invention locates specific bad-directed clips to support fine-grained traceability, generates a comprehensive chain of evidence for detecting bad-directed advertisements, realizes automatic detection of bad-directed illegal and irregular advertisements, accurately locates clues of bad-directed violations in advertisements, and assists manual interpretation.

[0055] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0056] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A fine-grained detection method for inappropriate content in advertising videos based on knowledge graphs and large language models, characterized by: The steps include: Step 1: Collecting advertising videos: Collect annotated videos of bad advertising videos from the illegal video advertising dataset published by regulatory authorities. This involves obtaining multimodal data including video streams, audio streams, subtitles, and regulatory comments. This data is then processed to generate a knowledge graph of bad advertising videos in the form of triples. Step 2: Video semantic segmentation. For videos longer than 30 seconds, two-stage video segmentation is performed: the first stage is to coarsely segment the video based on the shot switching to obtain n video segments [Video1, Video2, ..., Video n ], the second step is to fine-grainedly merge semantically similar video segments to obtain semantically complete video segments [Video 1’ ,Video 2’ ,...,Video n’ ], where n'≤n; Step 3: Extract and process audio emotion, visual, and text information; retrieve the associated violation pattern subgraph from the knowledge graph based on the advertisement type of the video to be tested; In step 4, the extracted information features and the knowledge graph subgraph from step 3 are input into the large model. The large model outputs the fine-grained analysis results of the bad video and generates a detection report.

2. The method for detecting inappropriate content in advertising videos based on a knowledge graph and a large language model according to claim 1 is characterized by: In step 1, the speech in the video is subjected to end-to-end automatic speech recognition technology to obtain text content, and the video frame is subjected to a visual language model to generate a natural language description. The combined text is cleaned, and meaningless interjections such as "hmm, uh" are filtered out based on regular expression matching. The text is input into the big model together with the video title text and regulatory commentary text. The big model extracts knowledge from the entities, relationships and attributes in the text by combining the predefined knowledge graph entities, relationships and attributes, and obtains the entity set E = {e1, e2, ..., e p } and the relation set R={(e i ,r ij ,e j )}, forming a knowledge graph S of bad-oriented advertising videos in the form of triples = { <e1,r1,e 1’> , <e2,r2,e 2’> ,..., <e n ,r n ,e n’ >}.

3. The fine-grained detection method for inappropriate content in advertising videos based on a knowledge graph and a large language model according to claim 2 is characterized by: In step 2, the advertisement video with a length of less than 30 seconds is not segmented.

4. The fine-grained detection method for inappropriate content in advertising videos based on a knowledge graph and a large language model according to claim 3 is characterized by: The step 3 is to process the audio of each video clip after segmentation, which is divided into three stages: audio processing → feature extraction → intonation classification. The human voice and background music in the advertisement are separated, and the acoustic features H = {h1, h2, ..., h n’ }, identify the intonation types through deep neural networks, obtain their probability distribution P(y|H), and select the intonation type V with the highest probability as the final judgment result.

5. The method for fine-grained detection of inappropriate content in advertising videos based on a knowledge graph and a large language model according to claim 4 is characterized by: At the visual level, step 3 may reveal undesirable cues about scenes or objects. A cascaded model of object detection and scene understanding is used to generate spatially annotated visual semantic pairs. Intermediate frames are obtained from each segmented video clip. The model detects products, people, and scene elements O, extracts image frames of these elements, and overlays them onto the original video frame I. A visual language model is then used to generate a textual description D = VLM(I,O) of the image.

6. The method for fine-grained detection of inappropriate content in advertising videos based on a knowledge graph and a large language model according to claim 5 is characterized by: In the text analysis of step 3, an open source speech-to-text model is used to extract the text content from the advertisement audio. Based on modifiers and comparative words with absolutist tendencies as clues to the negative orientation of the advertisement, a predefined absolutist vocabulary list and a comparative vocabulary list are pre-defined to select clue words such as "thoroughly, radically, compared to" as text features, which are expressed as T = {t1, t2, ..., t m }, m is the number of clue words.

7. The method for fine-grained detection of inappropriate content in advertising videos based on a knowledge graph and a large language model according to claim 6 is characterized by: The step 4 aligns the visual, textual, and auditory feature elements of step 3 with timestamps. Each video clip Video i All contain the characteristics {V i , D i , T i }, and input it into the large language model together with the relevant subgraphs of the retrieved knowledge graph.

8. The fine-grained detection method for inappropriate content in advertising videos based on a knowledge graph and a large language model according to claim 7 is characterized by: The process of obtaining the relevant subgraph of the knowledge graph is as follows: construct a retrieval vector q = Embedding based on the label of the advertising type to be tested, calculate its semantic similarity Sim(q,e) = cosine(q,Embedding(e)) with the predefined advertising type entity e∈E in the knowledge graph, and select the Top-5 similar entities; secondly, extract the 1-hop neighborhood subgraph of these entities in the knowledge graph through the graph traversal algorithm to form a triple set SubGraph = {(e,r,e')|e∈Top-5,e'∈N(e)} containing the violation pattern.

Images