Live broadcast goods carrying false propaganda detection method based on video, bullet screen and audio multi-mode information fusion
By employing multimodal feature extraction and cross-modal fusion methods, combined with knowledge graph-enhanced decision-making, the problems of low accuracy, poor real-time performance, and weak interpretability in live-streaming e-commerce false advertising detection have been solved, achieving efficient and real-time false advertising detection and interpretable evidence generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHAANXI SCI TECH UNIV
- Filing Date
- 2026-02-07
- Publication Date
- 2026-05-19
AI Technical Summary
Existing technologies for detecting false advertising in live-streaming e-commerce suffer from low accuracy, poor real-time performance, weak interpretability, and an inability to effectively integrate multimodal information from video, audio, and bullet comments.
Employing multimodal feature extraction, cross-modal fusion, and knowledge graph enhancement, this method utilizes temporal action recognition, streaming automatic speech recognition, pre-trained language models, and relational graph convolutional networks to achieve fine-grained feature extraction and cross-modal information fusion from video, audio, and bullet comments. Combined with knowledge graphs, it performs decision-level reasoning to generate interpretable false advertising detection results.
It achieves high-precision, real-time detection of false advertising, improves recall and accuracy, generates a clear multimodal evidence chain, facilitates manual review, and has strong scalability, adapting to live streaming platforms of different sizes.
Smart Images
Figure CN122065189A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence and digital content security technology, specifically relating to a method for detecting violations in live streaming content based on multimodal information fusion, and particularly a method and system for detecting false advertising in live streaming e-commerce by combining video, audio, and bullet screen data. Background Technology
[0002] In recent years, the live-streaming e-commerce industry has developed rapidly, but the accompanying problem of false advertising has become increasingly prominent. During live streams, hosts may make false or misleading claims about the performance, efficacy, price, and qualifications of products, seriously harming consumer rights and market order. Currently, the supervision of live-streaming content mainly relies on keyword filtering and manual review. Keyword filtering methods are easily circumvented by homophones and variations, and lack an understanding of the context; manual review is costly, inefficient, and unable to handle the massive, real-time volume of live streams.
[0003] Existing research often focuses on single-modal analysis, such as analyzing only post-live video recordings or only the text converted from ASR (Automatic Speech Recognition). These methods fragment the naturally occurring multimodal relationships between video (host behavior, product display), audio (promotional language, emotional tone), and bullet comments / comments (real-time consumer feedback) in a live streaming scenario. This results in low detection accuracy, high false positive rates, and an inability to provide real-time alerts. Furthermore, existing methods lack structured embedding of regulatory rules, making the decision-making process a "black box" with poor interpretability, failing to meet the requirements for complete evidence chains in actual regulation.
[0004] Therefore, there is an urgent need for a false advertising detection technology that can deeply integrate multimodal information from live streaming, provide real-time and accurate detection, and is interpretable. Summary of the Invention
[0005] (a) Purpose of the invention
[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for detecting false advertising in live-streaming e-commerce based on the fusion of multimodal information from video, bullet comments and audio, so as to solve the problems of low accuracy, poor real-time performance and weak interpretability of the existing detection technology.
[0007] (II) Technical Solution
[0008] To achieve the above objectives, the present invention provides the following technical solution:
[0009] A method for detecting false advertising in live-streaming e-commerce based on the fusion of multimodal information from video, bullet comments, and audio includes the following core steps:
[0010] 1. Multimodal feature extraction: Parallel analysis of video, audio, and bullet comments data in the live stream to extract fine-grained features.
[0011] Video analytics: Utilizing temporal action recognition models (such as SlowFast), the synchronization of the broadcaster's "demonstration-narration" actions is analyzed, and a synchronization score is calculated. Conduct a quantitative assessment: ,in, Let represent the video feature vector at time t. This represents the audio feature vector at time t. Represents the sigmoid function. The time window length for analysis is specified. Object detection models (such as YOLOv7) and OCR technologies (such as PaddleOCR) are used to identify product areas and price, brand, and certification label text in the image; micro-expression and gesture recognition models are used to capture the anchor's exaggerated expressions and misleading gestures.
[0012] Audio analysis: Speech is converted into text using a streaming automatic speech recognition model (such as Conformer); acoustic features such as Mel spectrum and fundamental frequency are extracted, and speech rate, pitch and emotional polarity are analyzed; preliminary matching of keywords for illegal speech is performed.
[0013] Bullet screen analysis: Bullet screens are aggregated in real time using streaming frameworks (such as Flink); fine-grained sentiment and intent analysis is performed using pre-trained language models (such as RoBERTa) to identify key intents such as "questioning" and "revealing"; high-frequency question and complaint words are mined through techniques such as TF-IDF; and abrupt changes in the frequency of bullet screen sending ("rhythm") are detected.
[0014] 2. Cross-modal fusion and alignment: Establish a unified timestamp system, using audio ASR time as the anchor point, to align video keyframes and bullet comment clusters on the timeline. A hierarchical fusion strategy is adopted.
[0015] Feature-level fusion: A cross-modal Transformer layer is designed to achieve cross-querying and information enhancement between video, audio, and text features through a multi-head attention mechanism. Video features (as the query) are used... ) and audio features (as keys) Sum Taking the interaction as an example, calculate the cross-attention from video to audio. : ,in, The dimension of the feature vector is used to scale the dot product result.
[0016] Semantic-level fusion: A dynamic multimodal heterogeneous graph is constructed, where nodes represent entities such as products, claims, questions, and sentiments, and edges represent semantic relationships between them. A relational graph convolutional network (RGCN) is used for message passing to learn a joint semantic representation that fuses multimodal contexts. For the first... Layer nodes Its characteristic representation The update formula is as follows: ,in, This represents the set of all relation types in the graph. Indicates a relationship Next and Node The set of connected neighbor nodes, The normalization constant is and For the first Layer-learnable weight matrix This represents a non-linear activation function.
[0017] Decision-level fusion: Introducing knowledge graphs to enhance reasoning.
[0018] 3. Knowledge Graph Enhanced Reasoning: A structured knowledge graph containing regulatory rules such as the Advertising Law and the Anti-Unfair Competition Law, as well as product domain knowledge, is pre-constructed. Nodes in the semantic graph output by the graph neural network are aligned with entities in the knowledge graph, triggering corresponding violation rules (such as "ordinary food - cannot claim - therapeutic function"), generating rule matching features and interpretable reasoning paths.
[0019] 4. False Advertising Identification and Classification: The semantic representation learned by the graph neural network is concatenated with the rule features obtained from knowledge graph reasoning, and then input into a multi-task classification model. This model simultaneously performs sequence labeling for violation time periods and fine-grained violation classification. Sequence labeling employs a Conditional Random Field (CRF) model, given an input sequence... Its output The probability is: ,in, It is a normalization factor. It is a characteristic function. These are the corresponding weights. Fine-grained classification uses a neural network with multi-head output layers, and its loss function... Using weighted cross-entropy:
[0020]
[0021] in, This represents the total number of violation categories. For real labels, To predict probabilities for the model, For categories The weighting coefficients are used to handle data imbalance problems.
[0022] The final output includes the type of violation, the time of occurrence, the confidence level, and related video, audio, and bullet screen evidence fragments.
[0023] (III) Beneficial Effects
[0024] Compared with the prior art, the present invention has the following significant advantages:
[0025] 1. High detection accuracy: It makes full use of the complementarity and mutual corroboration between multimodal information such as video, audio and bullet comments, and overcomes the ambiguity of single-modal analysis. Experimental verification shows that compared with single text analysis methods, this method improves the accuracy of false advertising detection by more than 15% and the recall rate by more than 12%.
[0026] 2. Strong real-time performance: It adopts a streaming processing architecture and parallel feature extraction, and optimizes the cross-modal alignment and fusion algorithm to achieve near real-time detection and early warning (under standard hardware configuration, the end-to-end average latency is 1.8 seconds, which meets the requirements of live streaming real-time supervision) and can meet the timeliness requirements of live streaming supervision.
[0027] 3. Good interpretability: By introducing knowledge graphs, the "black box" decision-making of deep learning models is combined with clear laws and regulations, which can output clear reasons for violation judgments and a complete "video-audio-bullet screen" multimodal evidence chain, greatly facilitating manual review and subsequent handling.
[0028] 4. Strong robustness: The multimodal fusion mechanism enables the system to make relatively reliable judgments based on information from other modalities even when the quality of a certain modal signal is poor (such as blurry images or noisy speech).
[0029] 5. High system scalability: The system adopts a microservice architecture design, with decoupled modules, which facilitates independent upgrades and horizontal expansion, and can adapt to the deployment needs of live streaming platforms of different sizes. Attached Figure Description
[0030] Figure 1 The overall system workflow diagram provided for embodiments of the present invention.
[0031] Figure 2 This is a system architecture diagram provided for an embodiment of the present invention.
[0032] Figure 3 This is a schematic diagram of the product detection and OCR process provided in an embodiment of the present invention.
[0033] Figure 4 This is a schematic diagram of the cross-modal attention fusion mechanism provided in an embodiment of the present invention.
[0034] Figure 5 This is a schematic diagram of the multimodal semantic graph structure provided in an embodiment of the present invention.
[0035] Figure 6 This is a flowchart of knowledge graph rule reasoning provided for an embodiment of the present invention.
[0036] Figure 7 The timing diagram of the real-time detection pipeline provided in the embodiment of the present invention.
[0037] Figure 8 This is a system deployment and monitoring architecture diagram provided for an embodiment of the present invention. Detailed Implementation
[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Figure 1 The system's overall workflow diagram shown illustrates that the method of this invention first accesses live stream data, then performs multimodal feature extraction, cross-modal fusion, and inference, ultimately outputting the false advertising detection result. Figure 2 The system architecture diagram shown illustrates that the system adopts a microservice layered architecture, which includes a data access layer, a feature calculation layer, a fusion inference layer, and a business application layer, ensuring the system's efficient operation and scalability.
[0039] Example 1: Implementation process of a method for detecting false advertising in live-streaming e-commerce.
[0040] This embodiment uses a live stream of a beauty product as an example to illustrate the implementation process of this method.
[0041] Step 1: Data Access and Splitting. The system accesses the live stream via the RTMP / FLV protocol and separates the video stream (H.264 encoded), audio stream (AAC encoded), and bullet comment stream (WebSocket protocol) in real time.
[0042] Step 2: Multimodal feature extraction is performed in parallel.
[0043] Video analysis: such as Figure 3 The diagram illustrates the product detection and OCR process, with the video stream processed at 30 frames per second. First, the product area is located using an object detection model, then region cropping and preprocessing are performed. Finally, OCR technology is used to recognize the text information on the packaging. The SlowFast model detected that the presenter was holding a face cream and demonstrating it at t=120s. YOLOv7 detected the face cream packaging in the frame, and PaddleOCR recognized the text "XX Face Cream," "Whitening and Spot Removing," and "Special Price ¥299." Simultaneously, the gesture recognition module detected that the presenter was making exaggerated comparison gestures regarding the product's size.
[0044] Audio Analysis: The audio stream for the corresponding time period was identified as text by the Conformer model: "This face cream, I guarantee that after using it for a week, all your blemishes will disappear, and your skin will be as white as a light bulb. It's absolutely the most powerful whitening product ever!" Acoustic analysis shows that the speech is fast-paced, high-pitched, and the emotional polarity is "extreme excitement." The keyword matching module triggered warning words such as "absolutely," "most powerful," and "guarantee."
[0045] Bullet Comment Analysis: Within the same time period, a large amount of text emerged in the bullet comment stream. The sentiment analysis model identified intentions such as "Really? (skepticism)" and "I used it and it didn't work" (exposure). TF-IDF analysis revealed high-frequency words such as "hormones" and "allergy". The bullet comment rate peaked at t=120s.
[0046] Step 3: Cross-modal alignment and fusion. The system uses the audio ASR timestamp as a reference to align the video keyframe (showing face cream) at t=120s with the bullet comment cluster (questioning and exposing) to form a multimodal segment unit.
[0047] Feature-level fusion: such as Figure 4 The diagram illustrates the cross-modal attention fusion mechanism. The cross-modal attention layer achieves feature interaction and enhancement through cross-attention calculations in three directions: video → audio, audio → text, and text → video. Specifically, it calculates the association weights between video features (face cream image) and audio-text features ("whitening," "blemish removal"), thereby strengthening the semantic representation of the face cream region in the video.
[0048] Semantic-level fusion: such as Figure 5 The diagram shows a multimodal semantic graph structure. A dynamic multimodal heterogeneous graph is constructed. Nodes are created: Product "XX Face Cream", Promotional Claim "Whitening and Spot-Fading, Effective in One Week", Consumer Question "Is it true or false?", Emotion "Anchor Excitement", "Comment Questioning". Edges are established: "Anchor - Claim - Claim", "Claim - Related - Product", "Consumer - Questioning - Claim".
[0049] Decision-level fusion and knowledge graph reasoning: Matching the nodes "XX face cream" (classified as "ordinary cosmetics") and "effective in one week" (involving "therapeutic efficacy claims") in the graph with a pre-built knowledge graph. For example... Figure 6 The flowchart shown illustrates the knowledge graph rule reasoning process. The system first performs entity alignment and linking, then queries the knowledge graph and performs rule matching, and finally constructs a chain of evidence. The knowledge graph contains the rule: "Ordinary cosmetics - cannot claim - medical efficacy". A successful match generates a rule violation feature. The knowledge graph matching process involves calculating the node representations in the semantic graph. With knowledge graph entity representation Cosine similarity between : When the similarity exceeds a preset threshold and the logical relationship violates the predefined prohibition rules in the knowledge graph (such as <ordinary cosmetics, prohibited claims, medical efficacy>), it is determined to be a rule trigger, and a high-dimensional rule violation feature vector is generated.
[0050] The real-time performance guarantee mechanism of the system processing flow, such as Figure 7 As shown in the diagram, this timeline illustrates the complete pipeline from batch input of video frames, audio clips, and bullet comments, to feature extraction, fusion inference, and final output. The system employs a sliding window processing mechanism, with each processing unit lasting 3 seconds and adjacent windows overlapping for 2 seconds to ensure continuous detection. Experiments show that, under standard hardware configuration, the system's end-to-end average latency is 1.8 seconds, with the feature extraction stage taking 0.5-1 second and the fusion inference stage taking 0.8-1.2 seconds, meeting the timeliness requirements of live streaming real-time monitoring.
[0051] Step 4: False Advertising Identification and Classification. The fused features are input into the classification model. The sequence labeling module determines the violation period to be from t=118s to t=125s. The classification module determines that it violates both Category I (false efficacy claims) and Category II (absolute terms) violations, with a combined confidence level of 0.92. Determined by multimodal evidence, the calculation formula is as follows: ,in, The original output probabilities of the classification model. This refers to the number of valid multimodal evidence pieces (such as specific gestures, infringing keywords, and questioning comments) supporting the current violation in the segment. The threshold for the amount of evidence. The harmonic coefficient (e.g., 0.7) represents the reconciliation factor between the model prediction and the supporting evidence. The system automatically associates and saves video clips, audio clips, and key bullet comments for that time period as evidence.
[0052] Step 5: Early Warning and Handling. Based on the preset confidence threshold strategy (low risk: 0.6-0.75, medium risk: 0.75-0.9, high risk: >0.9), the detection result has a confidence level of 0.92, which belongs to the high-risk violation level. A real-time alert is sent to the platform's review backend via API, and it is recommended to implement the "live broadcast interruption cooling-off period" action. At the same time, a detailed report is generated, including the violation type, timestamp, confidence level, violated rule clause (such as Article 28 of the Advertising Law), and multimodal evidence links.
[0053] Example 2: System Deployment Example
[0054] The system of the present invention can be adopted as follows Figure 8The system is deployed using a cloud-native architecture, as shown in the diagram. This diagram illustrates the containerized deployment scheme, including a distributed architecture of Pod Group 1 (feature extraction service), Pod Group 2 (fusion inference service), and Pod Group 3 (business service). The entire system is packaged as Docker containers and orchestrated and managed within a Kubernetes cluster. The data access layer, feature extraction layer, and fusion inference layer run as independent microservices, using Apache Kafka for message passing, Redis for caching intermediate features, and persisting results to a MySQL database. The monitoring system uses Prometheus to collect metrics (such as CPU / memory usage, detection latency, and accuracy) and Grafana for visualization. This architecture supports automatic elastic scaling based on the concurrency of the live stream, ensuring high availability.
[0055] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A method for detecting false advertising in live-streaming e-commerce based on the fusion of multimodal information from video, bullet comments, and audio, characterized in that, Includes the following steps: S1: Preprocess the live stream data to separate the video stream, audio stream, and bullet screen / comment stream; S2: Analyze and process the video stream to extract video features, including: extracting the synchronization features between the anchor's product display actions and the verbal content based on temporal action recognition; extracting product area features and text information features in the live broadcast based on target detection and optical character recognition; and extracting the anchor's micro-expression features and gesture features based on face recognition and pose estimation. S3: Analyze and process the audio stream to extract audio features, including: converting audio into text features based on automatic speech recognition, extracting speech rate, pitch and emotional polarity features based on acoustic feature analysis, and extracting features of illegal speech based on keyword matching; S4: Analyze and process the bullet screen / comment stream to extract text features, including: real-time aggregation of bullet screen cluster features based on a streaming text processing framework, extraction of features of questioning, revealing, and urging intentions based on a sentiment analysis model, extraction of features of high-frequency question words and complaint words of consumers based on keyword mining, and detection of bullet screen burst point features based on statistical analysis. S5: Establish a unified timestamp system, using the audio recognition time point as a benchmark, align video features and text features on the timeline to form multimodal segment units with temporal correlation; S6: Perform hierarchical fusion processing on the aligned multimodal features, including: S61. At the feature level, a cross-modal multi-head attention mechanism is used to achieve interaction and enhancement between video features, audio features, and text features; S62. At the semantic level, a dynamic multimodal heterogeneous graph is constructed. The nodes of the graph include product entities extracted from the video, promotional claims extracted from the audio, consumer questions extracted from the bullet comments, and emotional elements extracted from each modality. The edges of the graph represent the semantic relationships between the nodes. The graph is processed by a graph neural network for message passing and learning to obtain a joint semantic representation that integrates multimodal context. S63. At the decision level, the joint semantic representation output by the graph neural network is concatenated with the rule matching features obtained based on knowledge graph reasoning to form the final decision features; S7: Input the final decision features into the pre-trained classification model to identify and classify false advertising behavior in the live stream, and output the violation type, the time period of the violation, the confidence level, and the associated multimodal evidence fragments.
2. The method according to claim 1, characterized in that, In step S2, the synchronization feature is calculated as follows: Let the video feature sequence be where Given the time window length, the synchronicity score is... The calculation formula is: in, Let represent the video feature vector at time t. This represents the audio feature vector at time t. This represents the sigmoid function. The length of the analysis time window.
3. The method according to claim 1, characterized in that, In step S61, the cross-modal multi-head attention mechanism includes at least one of the following cross-attention calculations: (1) Video-to-audio cross-attention: using video features as the query matrix Using audio features as the key matrix Sum matrix ; (2) Audio-to-text cross-attention: using audio features as the query matrix Using text features as the key matrix Sum matrix ; (3) Text-to-video cross-attention: using text features as the query matrix Using video features as the key matrix Sum matrix . For example, using video features as a query. Using audio features as keys Sum Calculate the cross-attention from video to audio. : in, is the dimension of the feature vector.
4. The method according to claim 1, characterized in that, In step S62, the graph neural network is a relational graph convolutional network, and its first... Layer nodes Feature representation The update formula is: in, This represents the set of all relation types in the graph. Indicates a relationship Next and Node The set of connected neighbor nodes, The normalization constant is and For the first Layer-learnable weight matrix This represents a non-linear activation function.
5. The method according to claim 1, characterized in that, In step S63, the knowledge graph is a pre-built structured knowledge base, which uses relevant clauses from the Advertising Law and the Anti-Unfair Competition Law as its core rules to define prohibited associations between product categories, efficacy claims, and price descriptions; the rule matching features are generated in the following way: Perform semantic similarity matching between nodes related to products and claims in the dynamic multimodal heterogeneous graph and entities in the knowledge graph; If a match is successful and a prohibitive rule in the knowledge graph is triggered, the corresponding rule violation feature is generated, and the specific rule text and reasoning path are recorded.
6. The method according to claim 1, characterized in that, In step S7, the pre-trained classification model is a multi-task model, which includes: A sequence labeling module, using a conditional random field or Transformer decoder structure, is used to label the time series of live streams and accurately locate the start and end times of violations. A multi-label classification module employs a neural network structure that supports multi-label classification to perform fine-grained violation classification on the located time period. The violation categories include multiple predefined categories related to false advertising. The output of the classification model includes: violation category label, start and end timestamps of the violation time period, judgment confidence, and evidence index pointing to the original video clip, audio clip, and bullet screen clip.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.
9. A live-streaming e-commerce false advertising detection system based on the fusion of multimodal information from video, bullet comments, and audio, characterized in that... include: The data preprocessing module is used to separate the video, audio, and bullet screen data of the live stream; The multimodal feature extraction module is used to extract video features, audio features, and text features; The cross-modal fusion reasoning module is used for aligning, fusing, and reasoning about multimodal features from a knowledge graph. The false advertising detection module is used to output violation detection results; The early warning and response module is used to execute tiered response strategies.
10. The system according to claim 9, characterized in that, The system is built on a microservice architecture and deployed on a Kubernetes cluster using containerization technology. The modules communicate with each other through a message middleware. The warning and handling interface module performs different handling actions based on the threshold range set by the confidence level of the violation, including: recording and manual review when the confidence level is within the first threshold range; sending a reminder to the live broadcaster and implementing traffic limiting when the confidence level is within the second threshold range; and triggering live broadcast interruption and generating a violation report when the confidence level is higher than the third threshold.