Advertisement video unhealthy guidance fine-grained detection method based on knowledge graph and large language model

Abstract

The application relates to the fields of digital content security and artificial intelligence, and discloses an advertisement video unhealthy guidance fine-grained detection method based on a knowledge graph and a large language model, which comprises the following steps: 1, collecting advertisement videos to obtain a knowledge graph of unhealthy guidance advertisement videos in the form of triples after processing; 2, performing semantic segmentation on a to-be-detected advertisement video; 3, extracting and processing audio emotion, visual and text information, and retrieving an associated violation mode subgraph from the knowledge graph according to the advertisement type of the to-be-detected video; and 4, inputting the extracted information features and the knowledge graph subgraph of step 3 into a large model, outputting a fine-grained analysis result of the unhealthy video by the large model, and generating a detection report. The method of the application can locate specific unhealthy guidance segments to support fine-grained tracing, generate a comprehensive detection evidence chain of unhealthy guidance, realize automatic detection of unhealthy guidance illegal and irregular advertisements, accurately locate unhealthy guidance illegal clues in the advertisements, and assist manual interpretation.

Description

Technical Field

[0001] This invention relates to the fields of digital content security and artificial intelligence, specifically to a fine-grained method for detecting inappropriate content in advertising videos based on knowledge graphs and large language models. Background Technology

[0002] Existing advertising review technologies primarily focus on analyzing explicit violation clues in advertising videos, such as using rule engines based on keyword matching to filter out violating text. Unlike traditional illegal and non-compliant advertisements, the violations in misleading advertisements are implicit, mostly implied clues, and cannot be directly interpreted using legal provisions such as the Advertising Law. In actual regulatory work, manual screening is mainly relied upon, which cannot quickly match typical violation patterns of its type when new advertising videos are input, nor can it convert the violation elements of similar historical advertisements into inferable auxiliary clues. This results in time-consuming, labor-intensive, and inefficient methods. To address this, this application proposes a fine-grained detection method for misleading advertising videos based on knowledge graphs and large language models. Summary of the Invention

[0003] The purpose of this invention is to provide a fine-grained detection method for inappropriate advertising videos based on knowledge graphs and large language models, so as to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, this invention provides the following technical solution: a fine-grained detection method for inappropriate advertising videos based on knowledge graphs and large language models, comprising: Step 1, collection of advertising videos, collecting labeled inappropriate advertising video data from publicly available datasets of illegal video advertisements by regulatory authorities, obtaining multimodal data including video streams, audio streams, subtitle text, and regulatory comments, and processing it to obtain a knowledge graph of inappropriate advertising videos in the form of triples; Step 2, video semantic segmentation, performing two-stage video segmentation for videos longer than 30 seconds: the first stage coarse-grained segmentation of the video based on shot switching to obtain n video segments [Video1, Video2, ..., Video...]. n The second step involves fine-grained merging of semantically similar video segments to obtain semantically complete video segments. 1’ Video 2’ ,...,Video n’ ], where n'≤n; Step 3, extraction and processing of audio emotion, visual and text information; Step 4, inputting the extracted information features and the knowledge graph from Step 1 into the large model, the large model outputs fine-grained analysis results of bad videos, and generates a detection report.

[0005] Preferably, in step 1, the audio in the video is processed using end-to-end automatic speech recognition technology to obtain text content. The video frames are used to generate natural language descriptions through a visual language model. The combined text is then cleaned, and meaningless interjections such as "um" and "uh" are filtered out using regular expression matching. This text, along with the video title and regulatory commentary text, is input into a large model. The large model, combining predefined knowledge graph entities, relationships, and attributes, extracts knowledge from the entities, relationships, and attributes in the text, resulting in an entity set E = {e1, e2, ..., e...}. p} and relation set R = {(e i ,r ij ,e j )}, forming a knowledge graph S={ of negatively guiding advertising videos in the form of triplets <e1,r1,e 1’> , <e2,r2,e 2’> ,..., <e n ,r n ,e n’ >}.

[0006] Preferably, step 2 does not perform segmentation for advertising videos shorter than 30 seconds.

[0007] Preferably, step 3, for the audio processing of each segmented video clip, is divided into three stages: audio processing → feature extraction → intonation classification. This separates the human voice from the background music in the advertisement, and extracts acoustic features H = {h1, h2, ..., h} from the separated pure human voice. n’ The system identifies intonation types using a deep neural network, obtains their probability distribution P(y|H), and selects the intonation type V with the highest probability as the final judgment result.

[0008] As a preferred approach: Step 3, at the visual level, may contain undesirable directional cues from scenes or objects. A cascaded model of object detection and scene understanding is used to generate spatially labeled visual semantic pairs. Intermediate frames are obtained from each segmented video segment. The model detects product, person, and scene elements O, and image frames of these elements are extracted and superimposed onto the original video frame I. A visual language model is used to generate a text description D = VLM(I, O) for the image.

[0009] As a preferred approach: In step 3, the text analysis uses an open-source speech-to-text model to extract the text content from the advertising audio. Based on absolute modifiers and comparative words as clues to the advertising's negative message, a predefined absolute vocabulary and comparative vocabulary are used to select clue words such as "thoroughly," "cure," and "compared to" as text features, represented as T = {t1, t2, ..., t...}. m}, where m is the number of clue words.

[0010] Preferably, step 4 aligns the visual, textual, and auditory feature elements from step 3, which are time-stamped, for each video segment. i All contain the feature {V i D i T i}, and compare it with the knowledge graph related sub-data obtained in step 3. Figure 1 The input is fed into the large language model.

[0011] As a preferred approach, the process of obtaining the relevant subgraph of the knowledge graph is as follows: Based on the label of the advertising type to be tested, construct a retrieval vector q = Embedding(advertising type), calculate its semantic similarity Sim(q,e) = cosine(q,Embedding(e)) with the predefined advertising type entity e∈E in the knowledge graph, and select the Top-5 similar entities; then, extract the 1-hop neighborhood subgraph of these entities in the knowledge graph through the graph traversal algorithm to form a set of triples containing the violation pattern SubGraph = {(e,r,e')|e∈Top-5,e'∈N(e)}.

[0012] Compared with the prior art, the beneficial effects of this invention are as follows:

[0013] The method of this invention locates specific segments of misleading content to support fine-grained tracing, generates a comprehensive chain of evidence for detecting misleading content, realizes automatic detection of illegal and non-compliant advertisements with misleading content, accurately locates clues of misleading content and violations in advertisements, and assists in manual interpretation. Attached Figure Description

[0014] Figure 1 This is a flowchart of the method of the present invention;

[0015] Figure 2 This is a system framework diagram of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example

[0018] Please see Figures 1-2The diagram illustrates a fine-grained detection method for inappropriate advertising videos based on knowledge graphs and large language models. This method includes step 1: Collection of advertising videos. Annotated inappropriate advertising videos are collected from publicly available datasets of illegal video advertisements from regulatory authorities. Multimodal data containing video streams, audio streams, subtitles, and regulatory comments is obtained, and processed to generate a knowledge graph of inappropriate advertising videos in triplet form. Step 2: Video semantic segmentation. For videos longer than 30 seconds, a two-stage video segmentation process is performed: the first stage uses coarse-grained segmentation based on shot transitions to obtain n video segments [Video1, Video2, ..., Video...]. n The second step involves fine-grained merging of semantically similar video segments to obtain semantically complete video segments. 1’ Video 2’ ,...,Video n’ ], where n'≤n; Step 3, extraction and processing of audio sentiment, visual and text information, and retrieval of associated violation pattern subgraphs from the knowledge graph based on the advertising type of the video to be tested; Step 4, inputting the extracted information features and the relevant subgraphs from the knowledge graph retrieved in Step 3 into the large model, and the large model outputs fine-grained analysis results of the bad video and generates a detection report.

[0019] In this embodiment, to form an evidence chain for detecting potentially harmful content in advertising videos and to provide dynamic and interpretable decision support for compliance review of advertising content, it is necessary to extract advertising event units through video semantic segmentation, construct a multimodal feature set including visual scenes, voice emotion, and text semantics, and combine it with a domain knowledge graph to achieve collaborative detection of textual cues, visual metaphors, and intonation inducements, as well as cross-modal evidence complementarity. Causal reasoning is then performed using a large language model. The implementation process is as follows: Figure 1 As shown, it is divided into two main core modules:

[0020] The first part involves the construction of the knowledge graph: Annotated videos containing inappropriate content are collected from publicly available datasets of illegal video advertisements from regulatory authorities. Multimodal data, including video streams, audio streams, subtitles, and regulatory comments, is obtained. End-to-end automatic speech recognition technology is used to extract text content from the audio in the videos, and natural language descriptions are generated from the video frames using a visual language model. The combined text is then cleaned, and meaningless interjections such as "um" and "uh" are filtered out using regular expression matching. This text, along with the video title and regulatory comments, is input into a large model. The large model, combining predefined knowledge graph entities, relations, and attributes, extracts knowledge from the entities, relations, and attributes in the text, resulting in an entity set E = {e1, e2, ..., e...}. p} and relation set R = {(e i ,r ij ,e j)}, forming a knowledge graph S={ of negatively guiding advertising videos in the form of triplets <e1,r1,e 1’> , <e2,r2,e 2’> ,..., <e n ,r n ,e n’ >}.

[0021] The second part is the multimodal feature extraction of the advertisement videos to be tested: The advertisement videos to be tested vary in length, and different segments may have different semantics. For videos longer than 30 seconds, a two-stage video segmentation is performed: The first stage is based on coarse-grained segmentation of the video based on shot switching to obtain n video segments [Video1, Video2, ..., Video...]. n The second step involves fine-grained merging of semantically similar video segments to obtain semantically complete video segments. 1’ Video 2’ ,...,Video n’ ], where n'≤n. For advertising videos shorter than 30 seconds, no segmentation is performed. These videos contain multimodal information such as audio, images, and text, which reflect potential negative messages from different dimensions and require comprehensive analysis. The tone and intonation of the advertising video may be persuasive to viewers and could serve as important clues to negative messages. Audio processing for each segmented video clip is divided into three stages: audio processing → feature extraction → intonation classification. The human voice and background music in the advertisement are separated, and acoustic features H={h1,h2,…,h...} are extracted from the separated pure human voice. n’ The text describes a process for identifying tone types using deep neural networks, obtaining their probability distributions P(y|H), and selecting the tone type V with the highest probability as the final judgment. For the visual aspect, since scenes or objects may contain inappropriate cues, a cascaded model of object detection and scene understanding is used to generate spatially labeled visual semantic pairs. Intermediate frames are obtained from each segmented video segment, and the model detects product, person, and scene elements O. Image frames of these elements are extracted and superimposed onto the original video frame I. A visual language model is used to generate a text description D = VLM(I,O) for the image. For text analysis, an open-source speech-to-text model is used to extract the text content from the advertising audio. Considering that absolute modifiers and comparative words may serve as clues to inappropriate advertising, an absolute vocabulary and a comparative vocabulary are predefined. Clue words such as "thoroughly," "cure," and "compared to" are selected as text features, represented as T = {t1, t2, ..., t...}. m}, where m is the number of cue words, aligning visual, textual, and auditory feature elements with timestamps for each video clip. i All contain the feature {V i D iT i The process of obtaining the relevant subgraph of the knowledge graph is as follows: First, construct a retrieval vector q = Embedding(advertisement type) based on the label of the advertisement type to be tested. Calculate its semantic similarity Sim(q,e) = cosine(q,Embedding(e)) with the predefined advertisement type entity e∈E in the knowledge graph, and select the Top-5 similar entities. Second, extract the 1-hop neighborhood subgraph of these entities in the knowledge graph using a graph traversal algorithm, forming a set of triples containing the violation pattern SubGraph = {(e,r,e')|e∈Top-5,e'∈N(e)}. Then, compare this set with the retrieved relevant subgraph of the knowledge graph. Figure 1 The input is fed into a large language model to achieve fine-grained detection of inappropriate content in advertising videos. The complete detection method is as follows: Figure 2 As shown.

[0022] Furthermore, video data containing inappropriate advertising was collected from publicly available datasets of illegal videos from regulatory authorities. The audio portion was processed using a Whisper-large model to obtain text content, while the video portion was generated as a natural language description using a MiniCPM-V visual language model. The combined text underwent data cleaning, employing regular expression-based matching to filter out meaningless interjections such as "um" and "uh." The filtered text, along with the video title and regulatory commentary, was then input into a larger model. This model, combined with predefined entities, relationships, and attributes, extracted knowledge from the entities, relationships, and attributes in the text, resulting in an entity set E = {e1, e2, ..., e...}. p} and relation set R = {(e i ,r ij ,e j )}, forming a knowledge graph S={ of negatively guiding advertising videos in the form of triplets <e1,r1,e 1’> , <e2,r2,e 2’> ,..., <e n ,r n ,e n’ The predefined entities, relationships, and attributes are as follows:

[0023] Entity types include:

[0024] Advertising type (cosmetics / food / pharmaceuticals, etc.): Attributes include the publishing platform and the promoter's identity;

[0025] Product type (health products / cosmetics / financial products, etc.): Attributes include product name, target audience (elderly / young women / young men / children, etc.), and intended use;

[0026] Negative guidance type (feudal superstition / incitement of anxiety / gender discrimination, etc.): The attribute includes the degree of negativeness level (1-5);

[0027] The relationship includes:

[0028] The text contains an advertisement for a specific product.

[0029] The advertisement contains harmful content: The text contains advertising elements and keywords with harmful connotations.

[0030] Product-related negative connotations: Logical correlation between product usage and event type;

[0031] Input an advertisement video. For advertisement videos longer than 30 seconds, video segmentation is performed in two steps: The first step uses the PySceneDetect library in Python to segment the video based on camera transitions, obtaining n video segments [Video1, Video2, ..., Video...]. n The second step is to use the ImageBind model to extract each adjacent video segment. i 0.5 seconds before the end and Video i+1 The feature f of the first 0.5 seconds frame i and f i+1 If ||f i -f i+1 If ||≤threshold∈[0.8,1.0], then the two segments are concatenated into one segment, and the final segmentation yields semantically consistent video segments. Advertising videos shorter than 30 seconds are not segmented. For each segmented video segment, the open-source audio separation tool Spleeter is used to separate the human voice and background music in the advertisement. The separated pure human voice is then processed using the Wav2Vec2.0 model to extract acoustic features H={h1,h2,…,h…}, including pitch and spectrum. N}=Wav2Vec2.0(V vocal The algorithm identifies four tone types—urgent, intimidating, friendly, and authoritative—using a pre-trained deep neural network, obtaining their probability distributions P(y|H)=Softmax(W·DNN(H)+b), where W is the weight matrix, b is the bias vector, and y∈{urgent, fearful, friendly, authoritative}. The tone type V with the highest probability is selected as the final judgment. Three frames are uniformly taken from each video segment, and a YOLOv8s model is used to detect product, person, and scene elements. Image frames of these elements are extracted and superimposed onto the original video frames. The MiniCPM-V visual language model is used to generate text descriptions for the images, D=MiniCPM-V(I,O), where O is the extracted product, person, and scene element bounding boxes, and I is the original video frame. The Whisper-large model is used to extract the text content from the advertising audio: Text=Whisper(V vocalConsidering that absolute modifiers and comparative words may serve as clues to negative advertising, a predefined list of absolute terms and a list of comparative terms are used to filter out clue words such as "thoroughly," "cure," and "compared to," T = {t1, t2, ..., t...}. m}, where m is the number of cue words, aligning visual, textual, and auditory feature elements with timestamps for each video clip. i All contain the feature {V i D i T i The process of obtaining the relevant subgraph of the knowledge graph is as follows: A retrieval vector q = Embedding(advertisement type) is constructed based on the label of the advertisement type to be tested. The semantic similarity Sim(q,e) = cosine(q,Embedding(e)) between q and the predefined advertisement type entity e∈E in the knowledge graph is calculated, and the Top-5 similar entities are selected. Next, a graph traversal algorithm is used to extract the 1-hop neighborhood subgraph of these entities in the knowledge graph, forming a set of triples containing the violation pattern SubGraph = {(e,r,e')|e∈Top-5,e'∈N(e)}. This set is then compared with the retrieved relevant subgraph of the knowledge graph. Figure 1 The input is fed into a large language model to achieve fine-grained detection of inappropriate content in advertising videos.

[0032] Based on a knowledge graph, a thought process prompt is designed and fed into a large model to detect inappropriate content in advertising videos and generate a detection report, as shown in the example below.

[0033] As a professional advertising reviewer, prompt= can identify whether an advertisement contains harmful content. Harmful content in advertising includes creating appearance anxiety, misleading values, creating educational anxiety, involving feudal superstition, undermining group unity, hindering social stability, harming public interests, involving identity discrimination, harming physical and mental health, and promoting products prohibited by law. Based on the following advertising knowledge graph containing harmful content and the characteristics of the advertisement video, analyze and identify whether the advertisement video contains harmful content, and output the search results:

[0034] {Video_Features+Knowledge_Graph}

[0035] Need to be considered:

[0036] 1. Identify potential misleading elements in advertising videos and comprehensively scan for the following violation elements using timestamps:

[0037] Linguistic features: Does the text contain absolute terms or suggestive expressions?

[0038] Acoustic characteristics: Does the tone of voice have abnormal tones such as fearful / authoritative?

[0039] Visual features: Does the visual description contain suggestive image symbols or scenes?

[0040] If certain elements of violation exist, please analyze them in conjunction with the knowledge graph to determine whether they may contain harmful content.

[0041] 2. If no adverse guidance is found, provide the reasoning and output the result directly. If adverse guidance is found, provide a chain of evidence for the violation. The evidence chain format is [Violation Type Matching] → [Contradiction Analysis] → [Timestamp Association]. Combining multimodal evidence and knowledge graphs, the final result should be presented in a structured form.

[0042] (1) Clear identification of negative guidance types and credibility score (1-5)

[0043] (2) Detailed analysis of contradictions

[0044] (3) Accurate positioning of adverse guidance time periods down to the second.

[0045] Specifically, input: a cosmetics advertisement video;

[0046] Note: While promoting cosmetic products, the advertisement included a segment with the text "Fair skin = successful life" and a voiceover saying "My peers have already started changing their destiny through skin whitening."

[0047] Output (detection results):

[0048] "Type of violation": ["Spreading body image anxiety"]

[0049] "Spatiotemporal Positioning": [{"Start Time": "00:16.3", "End Time": "00:19.1",

[0050] "Evidence": ["Matched with historical violating entity <Quickly remove freckles, appearance anxiety, 4 points>", "Visual appearance of fair skin", "Presence of comparative language", "Urgent tone"]}]).

[0051] According to the categories of beauty, food, medicine, finance, education, games, online services, and tourism, 50 video advertisements were randomly collected in each category. The compliance test was carried out using the method proposed in this invention, and a manual inspection was used for re-inspection. The specific results are shown in Table 1 below.

[0052] Table 1 Compliance test results for different types of advertisements

[0053]

[0054] As can be seen from Table 1 above, the method proposed in this invention has a high accuracy rate in checking the compliance of different types of advertisements. The accuracy of machine inspection results is within 5% compared with the accuracy of manual inspection results. The method of this invention can locate specific segments with bad guidance to support fine-grained tracing, generate a comprehensive evidence chain for detecting bad guidance, realize the automatic detection of illegal and non-compliant advertisements with bad guidance, accurately locate clues of illegal and non-compliant behavior in advertisements, and assist manual interpretation.

[0055] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0056] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A fine-grained method for detecting inappropriate content in advertising videos based on knowledge graphs and large language models, characterized in that: Includes the following steps: Step 1: Collection of advertising videos. Collect labeled misleading advertising video data from the dataset of illegal video advertisements published by regulatory authorities. Obtain multimodal data including video stream, audio stream, subtitle text and regulatory comments. After processing, obtain a knowledge graph of misleading advertising videos in the form of triples. Step 2, video semantic segmentation: For videos longer than 30 seconds, a two-stage video segmentation process is performed: The first stage involves coarse-grained segmentation based on shot transitions to obtain n video segments [Video1, Video2, ..., Video...]. n The second step involves fine-grained merging of semantically similar video segments to obtain semantically complete video segments. 1’ Video 2’ ,...,Video n’ ], where n'≤n; Step 3: Extraction and processing of audio sentiment, visual and textual information; based on the advertising type of the video to be tested, retrieve the associated violation pattern subgraph from the knowledge graph; Step 4: Input the extracted information features and the knowledge graph subgraph from Step 3 into the large model. The large model outputs the fine-grained analysis results of the problematic videos and generates a detection report. Step 3, the audio processing for each segmented video clip, is divided into three stages: audio processing, feature extraction, and intonation classification. This involves separating the human voice from the background music in the advertisement and extracting acoustic features H={h1,h2,…,h} from the separated pure human voice. n’ The system identifies intonation types using a deep neural network, obtains their probability distribution P(y|H), and selects the intonation type V with the highest probability as the final judgment result. In step 3, at the visual level, there may be scenes or objects that have undesirable guiding implications. A cascaded model of object detection and scene understanding is used to generate visual semantic pairs with spatial annotations. Intermediate frames are obtained from each segmented video segment. The model detects product, person, and scene elements O, and the image frames of these elements are extracted and superimposed on the original video frame I. The visual language model is used to generate the text description of the image D=VLM(I,O). The text analysis in step 3 uses an open-source speech-to-text model to extract the text content from the advertisement audio. Based on absolute modifiers and comparative words as clues to the advertisement's negative message, a predefined absolute vocabulary and comparative vocabulary are used to select clue words as text features, represented as T={t1,t2,...,t...}. m }, where m is the number of clue words.

2. The fine-grained detection method for inappropriate advertising video content based on knowledge graphs and large language models according to claim 1, characterized in that: Step 1 involves using end-to-end automatic speech recognition technology to obtain text content from the audio in the video, and generating natural language descriptions from the video frames using a visual language model. The combined text is then cleaned, with meaningless interjections filtered out using regular expression matching. This cleaned text, along with the video title and regulatory commentary text, is input into a large model. The large model, combining predefined knowledge graph entities, relationships, and attributes, extracts knowledge from the entities, relationships, and attributes in the text, resulting in an entity set E={e1,e2,...,e...}. p } and relation set R={(e i ,r ij ,e j The knowledge graph S={ of negatively guiding advertising videos forms a triplet structure. <e1,r1,e 1’ >, <e2,r2,e 2’ >,..., <e n ,r n ,e n’ >}.

3. The fine-grained detection method for inappropriate advertising video content based on knowledge graphs and large language models according to claim 2, characterized in that: Step 2 does not perform segmentation on advertising videos shorter than 30 seconds.

4. The fine-grained detection method for inappropriate advertising video content based on knowledge graphs and large language models according to claim 1, characterized in that: Step 4 aligns the visual, textual, and auditory feature elements from Step 3, which are time-stamped, for each video segment. i All contain the feature {V i D i T i }, and input it together with the relevant subgraphs of the retrieved knowledge graph into the large language model.

5. The fine-grained detection method for inappropriate advertising video content based on knowledge graphs and large language models according to claim 4, characterized in that: The process of obtaining the relevant subgraph of the knowledge graph is as follows: Based on the advertising type tag to be tested, construct the retrieval vector q=Embedding, calculate its semantic similarity Sim(q,e)=cosine(q,Embedding(e)) with the predefined advertising type entity e∈E in the knowledge graph, and select the top-5 similar entities; then, extract the 1-hop neighborhood subgraph of these entities in the knowledge graph through the graph traversal algorithm to form a set of triples containing the violation pattern SubGraph={(e,r,e')|e∈Top-5,e'∈N(e)}.

Images