Method and system for detecting and tracing multi-mode false propaganda content of live e-commerce

Abstract

The invention discloses a detection and traceability method and system for live broadcast e-commerce multi-mode false propaganda content, belongs to the technical field of data processing, and aims to solve the technical problem that the live broadcast e-commerce false propaganda detection and traceability technology is insufficient. Comprising the steps of collecting text data, voice data and video data in a live broadcast process in real time; performing fine-grained alignment and fusion on the voice features, the video features and the text features through a cross-modal attention mechanism to obtain multi-modal features; constructing a large model based on a Transform model, taking the multi-modal features as input, performing false propaganda detection through the trained large model, and outputting a detection result including whether false propaganda exists or not, a violation type and confidence; performing graded early warning according to the confidence coefficient to form an intervention record; hash values of the original multi-modal data, the detection result and the intervention record are calculated, metadata are packaged, and the metadata are written into the block chain through an alliance chain PBFT consensus mechanism.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a live e-commerce multi-modal false propaganda content detection and tracing method and system. BACKGROUND

[0002] With the rapid development of live e-commerce, consumers increasingly rely on live product information to make purchase decisions. However, there are currently some unscrupulous merchants and anchors who use live streaming to engage in false propaganda, such as exaggerating product efficacy, faking sales, etc.

[0003] Existing detection methods have many shortcomings. On the one hand, many detection methods are based on single modal data, such as detecting only live text, but false propaganda may involve multiple aspects of content such as text, voice (intonation, emotion) and video (picture, product display), and single modal detection can easily miss key information, resulting in inaccurate detection results. On the other hand, existing technologies lack real-time capability and cannot timely detect false propaganda behavior during live streaming and handle it. In addition, when false propaganda behavior is detected, it is difficult to trace the entire live streaming process and accurately determine the source and transmission path of the violation, making it difficult for regulators and consumers to seek redress.

[0004] The deficiencies of live e-commerce false propaganda detection and tracing technology are technical problems that need to be solved. SUMMARY

[0005] The technical task of the present application is to provide a live e-commerce multi-modal false propaganda content detection and tracing method and system to solve the technical problem of the deficiencies of live e-commerce false propaganda detection and tracing technology.

[0006] In a first aspect, the present application provides a live e-commerce multi-modal false propaganda content detection and tracing method, characterized in that it comprises the following steps:

[0007] Data collection: real-time collection of multi-modal data during live streaming through a live streaming platform interface, and time stamp synchronization of multi-modal data, wherein the multi-modal data is unstructured data including text data, voice data and video data;

[0008] Feature extraction: voice analysis, feature extraction and emotion classification of voice data, outputting text with time stamp, acoustic features and emotion labels as voice features, target detection, product information verification, picture quality analysis and action expression analysis of video data, outputting target detection frame, product authenticity verification result, picture credibility score and action expression label as video features, semantic understanding and emotional intent analysis of text data, outputting entity-semantic relationship and emotional intent label as text features;

[0009] Multi-modal fusion: Through cross-modal attention mechanism, fine-grained alignment and fusion of speech features, video features and text features are performed to obtain multi-modal features, and dimension reduction operation is performed on the multi-modal features;

[0010] Modality detection: Based on the Transformer model, a large model is constructed, and the multi-modal features are input into the large model after training to detect false propaganda. The detection results include whether it is false propaganda, the type of violation and the confidence level;

[0011] Early warning intervention: According to the confidence level, a hierarchical early warning is performed. High-risk detection results are pushed to the regulatory department in real time and trigger the live pause instruction. Medium and low-risk detection results are recorded for review, and the early warning intervention operation is recorded to form an intervention record;

[0012] Evidence storage and traceability: The original multi-modal data, detection results and intervention records are calculated for hash values and encapsulated metadata, written into the blockchain through the PBFT consensus mechanism of the alliance chain, forming an unalterable full-link evidence to support two-way traceability of supervision and consumers.

[0013] As a preferred, during data collection, the speech data, video data and text data are synchronously collected through the inference API or pull stream interface opened by the live platform. The speech data is the audio stream of the real-time explanation of the anchor, the video data is the live picture stream, and the text data includes the real-time text appearing in the live picture and the comment data obtained from the live platform through the WebSocket interface. A uniform timestamp is added to all multi-modal data, and the discrete collected multi-modal data is cached and aligned through a sliding window.

[0014] As a preferred, the feature extraction includes the following operations:

[0015] For speech data, an end-to-end speech recognition model based on Transformer is used for real-time recognition to output timestamped text, and acoustic features of the speech are extracted. Based on the extracted acoustic features, a pre-trained speech emotion classification model is used for emotion label classification to output emotion labels. The acoustic features include Mel-frequency cepstral coefficients (MFCC), fundamental frequency (F0), energy contour (Energy Contour) and speech rate;

[0016] For video data, real-time target detection is performed on video frames by a YOLOv8 or DETR model to identify the anchor, the product display area, and the key action, the product packaging information is identified by OCR and combined with the product library to verify the authenticity of the product, the product packaging information includes the brand, model, and efficacy label of the product, whether the corresponding video of the video data exists a fake situation is detected, and a picture credibility score is output, the fake situation includes editing traces, filter transition use, and special effect forgery, the anchor's limb action is extracted by a pose estimation model, the anchor's expression is analyzed by a face recognition model, whether there is exaggerated performance behavior is judged, and an action expression label is output;

[0017] For text data, through data cleaning, advertising law prohibited words, repeated content, and irrelevant symbols are removed, and core propaganda techniques are retained, pre-trained language models are used to perform word segmentation, part-of-speech tagging, and named entity recognition on the cleaned text data, and key entities and semantic relationships are extracted, wherein the key entities include product name, efficacy promotion, and sales data, and corresponding examples of key entities and semantic relationships include product-efficacy-whitening, a sentiment classification model based on BERT fine-tuning is used to judge the sentiment tendency of the text data, and an intent classification model is used to identify whether there is an inducible technique, and a sentiment and intent label is generated.

[0018] As a preferred embodiment, the multi-modal fusion includes the following operations:

[0019] Text-speech alignment: aligning the timestamped text with the acoustic features of the original speech;

[0020] Video-text alignment: aligning the target detection box extracted from the video data with the target position mentioned in the text, and analyzing whether the actual effect in the video picture contradicts the text promotion;

[0021] Speech-video alignment: aligning the sentiment label based on speech data analysis with the action and limb based on video data analysis, and identifying whether there is a transition performance type of promotion;

[0022] Fusion feature dimension reduction: based on text-speech alignment, video-text alignment, and speech-video alignment, the fused features are generated, the dimension of the fused features is the dimension of the text features + the dimension of the speech features + the dimension of the video features, and the fused features are reduced to a predetermined dimension through a fully connected layer.

[0023] Preferably, the large model constructed based on the Transformer model comprises three single-modal encoders and a cross-modal fusion encoder, the three single-modal encoders are a text encoder, a speech encoder and a video encoder respectively, the text encoder is configured to perform self-attention encoding on text features, the speech encoder is configured to perform self-attention encoding on speech features, and the video encoder is configured to perform self-attention encoding on video features, and the cross-modal fusion encoder takes the outputs of the three single-modal encoders as inputs, is configured to learn inter-modal correlations through a multi-head attention mechanism and perform false propaganda detection, and outputs a detection result.

[0024] During training of the large model, the live broadcast data set is labeled, and training is performed through loss function minimization, the live broadcast data set comprises normal live broadcasts and false propaganda live broadcasts, the loss function adopts a multi-task learning loss, which comprises a false propaganda classification loss, a violation type classification loss and a multi-modal alignment loss, the false propaganda classification loss and the violation type classification loss are cross-entropy losses, and the multi-modal alignment loss is a contrast loss.

[0025] The trained large model is quantized and knowledge distilled before being deployed.

[0026] Preferably, the evidence storage and traceability comprises the following operations:

[0027] Data hash generation: calculating hash values of real-time collected original multi-modal data and detection results respectively to generate data hashes;

[0028] Data encapsulation: adding metadata to each data hash, wherein the metadata comprises basic information, a timestamp and an association identifier, the basic information comprises a live broadcast room number, an anchor ID, start / end time and a platform ID, the timestamp comprises original multi-modal data collection time and data hash generation time, and the association identifier is an association relationship of the multi-modal data;

[0029] On-chain storage: packaging the data hashes and the metadata into blocks, and writing the blocks into a blockchain through a PBFT consensus mechanism of a consortium chain;

[0030] Evidence verification: each block contains a hash value of a previous block, forming a chain structure, and a regulatory department queries the matching of a hash value of any block with original data through a blockchain browser, and the matching is verified by recalculating the hash value.

[0031] Preferably, the evidence storage and traceability comprises regulatory end traceability and consumer end traceability.

[0032] When the regulatory end traces the source, the supervisor queries the complete storage chain of the live broadcast through the blockchain node from the start of the live broadcast to the time period when the violation occurs, marks the data hash and detection results at each time point, obtains the original voice data, video data and text data from the storage system through the data hash, and verifies the consistency of the propaganda content and the actual goods;

[0033] When the consumer end traces the source, the blockchain storage two-dimensional code is embedded in the live broadcast page, the consumer scans the storage two-dimensional code to jump to the traceability page, the anchor basic information, risk prompt, key evidence chain and historical data comparison are displayed, the pre-determined time video clips before and after the violation time point in the key evidence chain and the corresponding voice text and detection results are selected, and downloading or notarization is supported, in the historical data comparison, the historical live broadcast data stored by the blockchain opens the API interface, the consumer retrieves the past live broadcast records through the product name and anchor ID, analyzes whether there is the mode of false propaganda of the same product multiple times or repeated use of similar rhetoric by the same anchor, and generates a risk report.

[0034] In a second aspect, the present application is a live broadcast e-commerce multi-modal false propaganda content detection and traceability system, which is used to realize the live broadcast e-commerce multi-modal false propaganda content detection and traceability method as described in any one of the first aspect, and the system comprises a data acquisition module, a feature extraction module, a multi-modal fusion module, a modal detection module, a early warning intervention module and a storage traceability module;

[0035] The data acquisition module is used to perform the following: real-time acquisition of multi-modal data in the live broadcast process through the live broadcast platform interface, and time stamp synchronization of the multi-modal data, wherein the multi-modal data is unstructured data, including text data, voice data and video data;

[0036] The feature extraction is used to perform the following: voice analysis, feature extraction and emotion classification on the voice data, and output of the text with time stamp, acoustic feature and emotion label as voice features, target detection, commodity information verification, picture quality analysis and action expression analysis on the video data, and output of the target object detection box, commodity authenticity verification result, picture credibility score and action expression label as video features, semantic understanding and emotional intent analysis on the text data, and output of the entity-semantic relationship and emotional intent label as text features;

[0037] The multi-modal fusion is used to perform the following: fine-grained alignment and fusion of the voice features, video features and text features through the cross-modal attention mechanism, to obtain multi-modal features, and dimension reduction operation on the multi-modal features;

[0038] The modal detection is used for performing the following: a large model is constructed based on a Transformer model, multi-modal features are taken as input, false advertising detection is performed through the large model after training, and detection results including whether false advertising, violation type and confidence are output;

[0039] The early warning intervention is used for performing the following: hierarchical early warning is performed according to the confidence, high-risk detection results are pushed to the supervision department in real time and a live broadcast pause instruction is triggered, medium and low-risk detection results are recorded for review, and early warning intervention operations are recorded to form intervention records;

[0040] The evidence storage and tracing is used for performing the following: the original multi-modal data, the detection results and the intervention records are respectively calculated for hash values and encapsulated for metadata, are written into a block chain through a PBFT consensus mechanism of an alliance chain, and form an unalterable full-link evidence, supporting two-way tracing of supervision and consumers.

[0041] The live broadcast e-commerce multi-modal false advertising content detection and tracing method system has the following advantages:

[0042] 1. Intervention reduces loss expansion: Traditional supervision relies on “after-the-fact spot checks”, and false advertising behavior often causes consumer losses (such as purchasing invalid goods or being induced to place orders). The present application can push early warnings to the supervision department in real time (delay < 500 ms) through the millisecond-level real-time detection of the multi-modal large model, and support automatic suspension of live broadcast (with the cooperation of the platform), shorten the supervision intervention from “several days after the event” to “seconds”, and effectively curb the spread of false information;

[0043] 2. Full-link evidence storage simplifies the evidence collection process: blockchain technology performs real-time hash chaining on live broadcast full-process data (voice, video, text, detection results), forming an unalterable “data fingerprint”. The supervision department does not need to retrieve scattered data (such as live broadcast recordings, chat records, and product links) across platforms, but can directly retrieve complete and reliable live broadcast full-link records through blockchain nodes, reducing the evidence collection time from the traditional “days” to “minutes”, and significantly reducing the law enforcement cost;

[0044] 3. Standardized evidence chain improves law enforcement credibility: The blockchain evidence storage data includes a unified space-time stamp, data source identifier and multi-modal correlation (such as a specific video image and voice tone at a certain time point corresponding to a false advertising phrase), avoiding disputes caused by multi-source data conflicts (such as platform deletion of part of the records or denial of the remarks by the host). The completeness and timeliness of the evidence chain are verified by the blockchain technology, which can be directly used as a judicial acceptance basis, improving the law enforcement efficiency and authority;

[0045] 4. Real-time risk prompt, auxiliary immediate decision: The violation results of the detection module are synchronized to the consumer end through the live platform interface (such as the live interface floating window, the product detail page label), and the risks such as "the current content may exaggerate the effect" and "the sales data is not verified" are prompted in simple language. For example, when the host claims that "this health product can cure diabetes", the consumer interface immediately pops up a red prompt "exaggerated effect detected, be careful of purchase", helping consumers quickly identify false propaganda and avoid impulsive consumption;

[0046] 5. Whole-process information traceability, supporting independent verification: Consumers can view the whole-link data summary of the live broadcast (such as product display pictures at key time points, host promise content, detection result records, etc.) by scanning the blockchain evidence two-dimensional code on the live broadcast page (or the traceability entry on the product detail page). For example, if the host claims "product sales 1 million+", consumers can compare the "background sales screenshot" displayed in the live broadcast with the original data in the blockchain evidence through the traceability page to verify its authenticity; if the picture is blurred or the screenshot is tampered with, the data integrity can be directly verified through the blockchain hash value;

[0047] 6. Historical data comparison, identify repeated frauds The historical live broadcast data stored in the blockchain is open to consumer query interface, supporting search of past live broadcast records by product name and host ID. For example, consumers can query "whether a host has used the "limited time sale" tactic multiple times in the past six months" and "whether the same product has been repeatedly faked in sales", and the system automatically analyzes abnormal patterns and generates a risk report to reduce the probability of repeated fraud;

[0048] 7. Multi-modal fusion to compensate for single-modal defects: Traditional methods are based on text or voice detection only, which may miss picture contradictions in the video (such as the host claiming "fresh fruits" but the picture showing moldy fruits) or emotional inducement in the voice (such as excessive "promotion" language). This invention uses cross-modal attention mechanism (such as correlation analysis of text "top sales" and video "background data screenshot", and semantic binding of voice "excited tone" and text "exaggerated promise"), to achieve "multi-dimensional cross verification" of false propaganda, with detection accuracy improved by more than 30% compared to single-modal methods (as verified by experiments, among the 100,000 test data labeled, the F1 score of multi-modal detection reached 0.92, while single-modal text detection was only 0.65);

[0049] 8. Large model learns complex correlations and adapts to various scenarios: Based on the multi-modal large model of Transformer, the self-attention and cross-modal attention mechanism can learn the implicit correlation between different modalities (such as the correlation between the "eye avoidance" action of the anchor and the "false promise" text, and the correlation between the overuse of filters in the picture and the "exaggerated effect" voice), and can identify hidden false propaganda that traditional rule engines cannot cover (such as "suggestion of therapeutic effect" and "ambiguous sales expression"). Experiments show that the model's recognition accuracy for "fake experimental effect" (such as the anchor demonstrating "instant whitening of skin care products" but the actual picture is a clip) reaches 0.89, which is significantly higher than the traditional rule method (0.52);

[0050] 9. Traditional evidence storage methods (such as platform self-recording and broadcasting) have the risk of data being tampered with and deleted, making it difficult for regulators to gather evidence. Through the distributed storage and hash chain structure of the blockchain, the immutability of live streaming data throughout the process is ensured: the SHA-256 hash value of each piece of data (voice, video, text, detection result) is calculated and chained, and any modification of the original data will cause the hash value to change, which can be verified by recalculating the hash value to verify the integrity of the data. The PBFT consensus mechanism of the blockchain (the alliance chain is jointly maintained by the regulatory department, the platform, and the notary agency) ensures that data writing needs to be verified by multiple parties to prevent single-node tampering; the evidence storage data includes timestamps, data source identifiers, and cross-modal correlation relationships, forming a complete evidence chain that is "traceable, traceable, and accountable", providing "iron evidence" support for regulators and consumers;

[0051] 10. Through technical means, a closed-loop management mechanism of "detection-early warning-tracing" is constructed, which on the one hand forces anchors and merchants to standardize their promotional behavior (as illegal behavior can be detected and traced in real time), and on the other hand improves consumers' trust in live streaming shopping (by reducing the risk of being cheated through risk warnings and self-verification), which helps to promote the transformation of live streaming e-commerce from "traffic-driven" to "trust-driven" in the long run, and promotes the healthy and sustainable development of the entire industry. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creating any inventive labor.

[0053] The present application will be further described below in conjunction with the drawings.

[0054] Figure 1 The flowchart of the live streaming e-commerce multi-modal false propaganda content detection and tracing method of embodiment 1. DETAILED DESCRIPTION

[0055] The application will be further described below in conjunction with the drawings and specific embodiments so that those skilled in the art can better understand and implement the application, but the embodiments are not intended to limit the application. The embodiments of the application and the technical features in the embodiments can be combined with each other without conflict, if possible.

[0056] The embodiment of the application provides a live e-commerce multi-modal false advertising content detection and tracing method and system, which is used to solve the technical problems of live e-commerce false advertising detection and tracing technology.

[0057] Embodiment 1:

[0058] The live e-commerce multi-modal false advertising content detection and tracing method provided by the application comprises six steps of data acquisition, feature extraction, multi-modal fusion, modal detection, early warning intervention and evidence storage and tracing.

[0059] Step S100 data acquisition: multi-modal data in the live process is collected in real time through a live platform interface, and the multi-modal data is time-stamped synchronously, wherein the multi-modal data is unstructured data, including text data, voice data and video data.

[0060] As a specific implementation, when data is collected, voice data, video data and text data are synchronously collected through an inference API or a stream pulling interface opened by the live platform, wherein the voice data is an audio stream of real-time explanation of an anchor, the video data is a live picture stream, the text data includes real-time text appearing in the live picture and comment / correction data obtained from the live platform through an OCR tool for recognizing a character area in the live picture, and a uniform time stamp is added to all the multi-modal data, and the discrete collected multi-modal data is cached and aligned through a sliding window.

[0061] In this step, voice data, video data and text data are synchronously collected, the voice data is an audio stream of real-time explanation of an anchor (sampling rate 16 kHz, single channel, encoding format PCM / AAC), the video data is a live picture stream (resolution 1080P / 720P, frame rate 25 fps / 30 fps, encoding format H.264 / H.265), and the text data is real-time text appearing in the live process (including OCR recognition results of anchor voice-over scripts, audience comment data, interactive text in a comment area, product link titles / descriptions, etc.), the character area in the live picture is recognized in real time through an OCR tool (such as Tesseract or Baidu OCR), and comment / correction data pushed by the platform is obtained through a WebSocket interface.

[0062] Timestamp synchronization is performed on the collected multi-modal data, a unified timestamp (millisecond-level timestamp calibrated based on an NTP server) is added to all modal data, and voice, video, and text data are aligned in the time dimension (for example, the sending time of a certain barrage is consistent with the frame timestamp of the anchor's speaking in the video). The discrete collected multi-modal data streams (such as voice stream slicing by 1 second, video stream frame acquisition, and text stream event triggering) are cached and aligned through a sliding window (window size 5 seconds) to solve the misalignment problem caused by data transmission delay.

[0063] Step S200 feature extraction: voice analysis, feature extraction, and emotion classification are performed on the voice data, and timestamped text, acoustic features, and emotion labels are output as voice features. Target detection, product information verification, picture quality analysis, and action expression analysis are performed on the video data, and target detection frame, product authenticity verification result, picture credibility score, and action expression label are output as video features. Semantic understanding and emotional intent analysis are performed on the text data, and entity-semantic relationship and emotional intent label are output as text features.

[0064] As a specific implementation of feature extraction, this step includes the following operations:

[0065] (1) For voice data, real-time recognition is performed through a Transformer-based end-to-end speech recognition model, timestamped text is output, and acoustic features of the voice are extracted. Based on the extracted acoustic features, an emotion label is classified through a pre-trained voice emotion classification model, and the emotion label is output. The acoustic features include Mel-frequency cepstral coefficients (MFCC), fundamental frequency (F0), energy contour, and speech rate.

[0066] (2) For video data, real-time target detection is performed on the video frame through a YOLOv8 or DETR model to identify the anchor, product display area, and key action. The product packaging information is identified through OCR and combined with the product library to verify the authenticity of the product. The product packaging information includes the brand, model, and efficacy label of the product. It is detected whether the video data has a fake situation, and a picture credibility score is output. The fake situation includes editing traces, filter transition use, and special effect forgery. The anchor's body action is extracted through a pose estimation model, the anchor's expression is analyzed through a face recognition model, it is judged whether there is exaggerated performance behavior, and an action expression label is output.

[0067] (3) For text data, remove illegal words, repeated content, and irrelevant symbols through data cleaning, retain core promotional phrases, perform word segmentation, part-of-speech tagging, and named entity recognition on the cleaned text data through a pre-trained language model, extract key entities and semantic relationships, including product name, efficacy promotion, and sales data, and corresponding examples of key entities and semantic relationships include product-efficacy-whitening, determine the sentiment of the text data based on a BERT fine-tuned sentiment classification model, and identify whether there are inducement phrases through an intent classification model to generate sentiment and intent labels.

[0068] This step converts unstructured multi-modal data into structured features that can be processed by the model, while filtering out noise interference.

[0069] Voice data processing and feature extraction includes the following operations:

[0070] (1) Speech to text (ASR): Use a Transformer-based end-to-end speech recognition model (such as Wav2Vec2.0 or Conformer) to recognize 1-second speech streams in real time, outputting timestamped text (such as "00:01:23-00:01:25 anchor says: this mask has 3 times the moisturizing effect of ordinary masks");

[0071] (2) Acoustic feature extraction: Extract Mel-frequency cepstral coefficients (MFCC), fundamental frequency (F0), energy contour, speech rate (words per minute), and other features of the voice, which can be used to analyze the anchor's tone (such as excited, flat) and emotions (such as excited, anxious);

[0072] (3) Sentiment classification: Use a pre-trained voice sentiment classification model (such as a CNN-LSTM-based model) to classify voice segments into sentiment labels (such as "exaggerated", "induced", "neutral"), which can help determine the direction of false propaganda.

[0073] Video data processing and feature extraction includes the following operations:

[0074] (1) Object detection and segmentation: Use YOLOv8 or DETR models to perform real-time object detection on video frames, identify anchors, product display areas (such as products placed on the table, experimental props), and key actions (such as "pick up product display" "apply skin care products");

[0075] (2) Product information extraction: Use OCR to identify brand, model, and efficacy labels (such as "whitening and fading spots" "24-hour moisturizing") on product packaging, and verify product authenticity by combining with a product library (previously connected to an e-commerce platform product database);

[0076] (3) Picture quality analysis: Detect whether there are video editing traces (such as picture frame skipping, repeated segments), excessive use of filters (such as skin smoothing that makes skin imperfections invisible), and special effects forgery (such as virtual like number animation), and output picture credibility score;

[0077] (4) Action and expression analysis: Extract the host's body movements (such as hand gesture amplitude and frequency) through a pose estimation model (such as OpenPose), analyze expressions (such as excessive smiling and eye avoidance) through a facial expression recognition model (such as an expression classification model based on ResNet-50), and determine whether there are exaggerated performance behaviors.

[0078] Text data processing and feature extraction includes the following operations:

[0079] (1) Text cleaning: Remove advertising banned words (such as "best", "first", "absolute"), repetitive content, and irrelevant symbols (such as emoticons and links), and retain core promotional language;

[0080] (2) Semantic understanding: Perform word segmentation, part-of-speech tagging, and named entity recognition (NER) on the cleaned text through a pre-trained language model (such as BERT-base-Chinese), extract key entities (such as product name, efficacy claim, and sales data), and semantic relationships (such as "Product A → Efficacy → Whitening");

[0081] (3) Sentiment and intent analysis: Based on a BERT fine-tuned sentiment classification model, determine the text sentiment (such as "exaggerated", "false promise", and "neutral"), and through an intent classification model, identify whether there are inducement tactics (such as "limited-time flash sale" and "only 10 left").

[0082] Step S300 multi-modal fusion: Through cross-modal attention mechanism, fine-grained alignment and fusion of speech features, video features, and text features are performed to obtain multi-modal features, and dimensionality reduction operation is performed on the multi-modal features.

[0083] As a specific implementation of multi-modal fusion, this step includes the following operations:

[0084] (1) Text-speech alignment: Align the timestamped text with the acoustic features of the original speech;

[0085] (2) Video-text alignment: Align the target object detection box extracted from the video data with the target object position mentioned in the text, and analyze whether the actual effect in the video picture contradicts the text promotion;

[0086] (3) Speech-video alignment: Align the sentiment labels based on speech data analysis with the body movements based on video data analysis, and identify whether there are performance-style promotions;

[0087] (4) Fusion feature dimension reduction: generate the fused feature based on text-speech alignment, video-text alignment and speech-video alignment, the dimension of the fused feature is text feature dimension + speech feature dimension + video feature dimension, and the fused feature is reduced to a predetermined dimension through a fully connected layer.

[0088] This step of the embodiment determines whether the live broadcast content contains false propaganda through multi-modal information interaction and locates the type of violation. The specific implementation is as follows:

[0089] (1) Text-speech alignment: align the text output by the ASR with the intonation / emotion features of the original speech, for example, when “sales of 1 million +” appears in the text, whether excited intonation or repeated emphasis is accompanied in the attention speech;

[0090] (2) Video-text alignment: align the target detection result (such as “commodity A”) of the commodity display screen with the “commodity A” mention position in the text, analyze whether the actual effect (such as “the whitening effect is not obvious”) of the commodity in the screen is contradictory to the text claim (such as “7-day whitening”);

[0091] (3) Speech-video alignment: align the speech emotion (such as “excited”) of the anchor with the action expression (such as “hand dancing”) in the video, and identify whether there is “excessive performance propaganda”.

[0092] The dimension of the fused feature vector is [text feature dimension + speech feature dimension + video feature dimension], which is reduced to a fixed dimension (such as 512 dimensions) through a fully connected layer as the input of the detection model.

[0093] Step S400 modality detection: a large model is constructed based on a Transformer model, multi-modal features are input, false propaganda detection is performed through the large model after training, and detection results including whether false propaganda, violation type and confidence are output.

[0094] The large model constructed based on the Transformer model in the embodiment includes three single-modal encoders and a cross-modal fusion encoder. The three single-modal encoders are a text encoder, a speech encoder and a video encoder. The text encoder is used for self-attention encoding of text features, the speech encoder is used for self-attention encoding of speech features, and the video encoder is used for self-attention encoding of video features. The cross-modal fusion encoder takes the outputs of the three single-modal encoders as input, learns the inter-modal association through a multi-head attention mechanism, and performs false propaganda detection to output the detection results.

[0095] Wherein the large model is trained based on the labeled live data set, through loss function minimization, the live data set includes normal live and false propaganda live, the loss function adopts multi-task learning loss, including false propaganda classification loss, violation type classification loss and multi-modal alignment loss, the false propaganda classification loss and the violation type classification loss are cross entropy loss, and the multi-modal alignment loss is contrast loss.

[0096] The large model after training is quantized and knowledge distilled and then deployed.

[0097] The model architecture in the embodiment: a multi-modal encoder based on Transformer, including three single-modal encoders (text encoder, speech encoder and video encoder) and a cross-modal fusion encoder.

[0098] Single-modal encoder: self-attention encoding of text, speech and video features respectively, capturing intra-modal semantic association.

[0099] Cross-modal fusion encoder: learning inter-modal association through multi-head attention mechanism (Multi-Head Attention), for example, the relevance between the text "top sales" and the "background sales data screenshot" in the video.

[0100] Training data: using a labeled live data set (containing 100,000 samples), of which 50,000 are normal live (without false propaganda) and 50,000 are false propaganda (annotated violation types: exaggeration, fake sales, false promise, fake experimental effect, etc.).

[0101] Loss function: multi-task learning loss is adopted, including:

[0102] False propaganda classification loss (cross entropy loss, predicting whether it is false propaganda);

[0103] Violation type classification loss (cross entropy loss, predicting the specific violation type);

[0104] Multi-modal alignment loss (contrast loss, narrowing the similarity of relevant modal features and pushing away irrelevant features);

[0105] Real-time inference optimization: through model quantization (FP32→INT8), knowledge distillation (compressing the large model into a lightweight student model) and TensorRT acceleration, the inference delay on the GPU server is ≤500ms, meeting the real-time requirement (live picture frame processing delay ≤200ms).

[0106] Step S500 early warning intervention: hierarchical early warning according to confidence, high-risk detection results are pushed to the regulatory department in real time and trigger live pause instructions, medium and low-risk detection results are recorded for review, and early warning intervention operations are recorded to form intervention records.

[0107] This step of the embodiment realizes the closed-loop management of "detection-early warning-disposal", improving the supervision efficiency.

[0108] Hierarchical early warning: according to the confidence (such as 0-1 points) of the detection model, the violation risk is divided into three levels:

[0109] High risk (confidence ≥ 0.9): immediately trigger early warning, push pop-up window notice (including violation timestamp, type, live segment link) to the regulatory department, and automatically pause live (need platform cooperation);

[0110] Medium risk (0.7 ≤ confidence < 0.9): push early warning information to the regulatory department for manual review;

[0111] Low risk (confidence < 0.7): record to the log for subsequent analysis and optimization of the model.

[0112] Cross-department cooperation: through the permission management function of the blockchain, different query permissions are allocated to the market supervision department, the public security department and the consumer association, ensuring data security while realizing collaborative law enforcement (such as the market supervision department calling commodity qualification data, and the public security department checking the criminal responsibility of false propaganda).

[0113] Step S600 evidence storage and traceability: calculate the hash value of the original multi-modal data, detection results and intervention records respectively and encapsulate the metadata, write into the blockchain through the PBFT consensus mechanism of the alliance chain, form an unalterable full-link evidence, and support two-way traceability of supervision and consumers.

[0114] As a specific implementation of evidence storage and traceability, this step includes the following operations:

[0115] (1) Data hash generation: calculate the hash value of the real-time collected original multi-modal data and detection results respectively to generate data hash;

[0116] (2) Data encapsulation: add metadata to each data hash, wherein the metadata includes basic information, timestamp and association identifier, the basic information includes live room number, anchor ID, start / end time and platform ID, the timestamp includes original multi-modal data collection time and data hash generation time, and the association identifier is the association relationship of multi-modal data;

[0117] (3) On-chain storage: package the data hash and metadata into a block, and write it into the blockchain through the PBFT consensus mechanism of the alliance chain;

[0118] (4) Storage verification: Each block contains the hash value of the previous block, forming a chain structure, and the regulatory department queries the matching of any block hash value and original data through the block chain browser, and verifies by recalculating the hash value when calculating the matching.

[0119] As a specific implementation of storage traceability, it can realize supervision end traceability and consumer end traceability.

[0120] When the regulatory end traceability, the regulatory personnel query the complete storage chain of live broadcast through the block chain node, from the time period from the beginning of live broadcast to the occurrence of violation, label the data hash and detection result of each time point, obtain the original voice data, video data and text data from the storage system through the data hash, and verify the consistency of the propaganda content and the actual goods.

[0121] When the consumer end traceability, the blockchain storage two-dimensional code is embedded in the live broadcast page, the consumer scans the storage two-dimensional code and jumps to the traceability page, displays the anchor basic information, risk prompt, key evidence chain and historical data comparison, selects the pre-determined time video segment of the violation time point and the corresponding voice text and detection result in the key evidence chain, and supports downloading or notarization, when the historical data comparison, the historical live broadcast data stored by the block chain opens the API interface, the consumer retrieves the past live broadcast record through the product name and anchor ID, analyzes whether there is the same product multiple false propaganda or the same anchor repeatedly uses similar mode of words, and generates a risk report.

[0122] In this embodiment, the live broadcast whole process data is stored in an unalterable manner, and the regulatory and consumer can quickly trace the violation behavior.

[0123] Data hash generation: The hash value (SHA-256 algorithm) of the real-time collected multi-modal original data (voice, video, text) and detection result (violation timestamp, violation type, model confidence) is calculated respectively to generate data hash.

[0124] Metadata encapsulation: Add metadata to each data hash, including: basic information: live broadcast room number, anchor ID, start / end time, platform ID; timestamp: data collection time (NTP synchronization), hash generation time; association identifier: association relationship of cross-modal data (such as video frame timestamp and voice segment ID corresponding to a certain text words).

[0125] On-chain storage: Pack the data hash + metadata into a block, and write it into the block chain through the PBFT (practical Byzantine fault tolerance) consensus mechanism of the alliance chain (maintained by the regulatory department, live broadcast platform and third-party notarization agency).

[0126] Evidence verification: Each block contains the hash value of the previous block, forming a chain structure to ensure data cannot be tampered with; regulatory authorities can query the matching of the hash value and the original data of any block through the blockchain browser (verified by recalculating the hash value).

[0127] Full-link traceability implementation includes regulatory traceability and consumer end traceability.

[0128] Regulatory end traceability: When false propaganda is detected, regulatory personnel can query the complete evidence chain of the live broadcast through the blockchain node, including: key time axis: timeline from the start of the live broadcast to the occurrence of the violation, marking the multi-modal data hash, detection result at each time point; original data retrieval: obtain the original voice, video, and text data from the evidence storage system (such as IPFS distributed storage) through the hash value to verify the consistency of the propaganda content and the actual product.

[0129] Consumer end traceability: Embed the blockchain evidence QR code in the live broadcast page, and after the consumer scans it, jump to the traceability page to display: live broadcast basic information (anchor, product, time); risk prompt (such as "detected exaggeration of efficacy, violation time 00:15:30"); key evidence chain: select 5-second video clips before and after the violation time point, corresponding voice text, and detection result details, support download or notarization; historical data comparison: open API interface for historical live broadcast data stored in the blockchain, consumers can retrieve past live broadcast records by product name or anchor ID, and the system automatically analyzes whether there are patterns such as "multiple false propaganda of the same product" and "similar rhetoric repeated use by the same anchor", and generates a risk report.

[0130] The method of the embodiment collects multi-modal data (text, voice, video) in real time through the live broadcast platform interface, and synchronizes the data with timestamps and pre-processes it to extract text, voice, and video features; uses a cross-modal attention mechanism to fuse multi-modal features to generate a fusion vector, which is input into a multi-modal large model (based on the Transformer architecture, including a single-modal encoder and a cross-modal fusion encoder) to output false propaganda detection results (including violation type and confidence); hashes the multi-modal original data and detection results to encapsulate metadata, which is written into the blockchain through a consortium chain (PBFT consensus mechanism) to form an unalterable full-link evidence.

[0131] This method realizes the "real-time discovery-accurate judgment-trustworthy traceability" closed loop of live e-commerce false propaganda through the collaborative design of multi-modal real-time detection and blockchain traceability, providing government departments with an efficient regulatory tool for "second-level early warning-minute-level evidence collection", and creating a fraud prevention assistant for consumers with "risk prompt-self-verification-historical traceability", significantly improving the information transparency and trustworthiness in the live e-commerce field.

[0132] Embodiment 2:

[0133] The application discloses a live e-commerce multi-modal false propaganda content detection and tracing system, which comprises a data acquisition module, a feature extraction module, a multi-modal fusion module, a modal detection module, a pre-warning intervention module and a storage and tracing module.

[0134] The data acquisition module is configured to perform the following: collecting multi-modal data in a live process in real time through a live platform interface, and synchronizing time stamps for the multi-modal data, wherein the multi-modal data is unstructured data, including text data, voice data and video data.

[0135] As a specific implementation, the data acquisition module is configured to synchronously collect voice data, video data and text data through an inference API or a stream pulling interface opened by the live platform, wherein the voice data is an audio stream of real-time explanation by an anchor, the video data is a live picture stream, the text data includes real-time text appearing in the live picture and comment / criticism data obtained from the live platform through an OCR tool for recognizing a character region in the live picture, a uniform time stamp is added to all the multi-modal data, and discrete collected multi-modal data is cached and aligned through a sliding window.

[0136] The feature extraction module is configured to perform the following: performing voice analysis, feature extraction and emotion classification on the voice data, outputting text with a time stamp, acoustic features and emotion labels as voice features, performing target detection, commodity information verification, picture quality analysis and action expression analysis on the video data, outputting target object detection frames, commodity authenticity verification results, picture credibility scores and action expression labels as video features, performing semantic understanding and emotional intent analysis on the text data, and outputting entity-semantic relationships and emotional intent labels as text features.

[0137] As a specific implementation of the feature extraction module, the module is configured to perform the following operations:

[0138] (1) for the voice data, real-time recognition is performed through an end-to-end voice recognition model based on a Transformer, text with a time stamp is outputted, acoustic features of the voice are extracted, emotion labels are classified based on the extracted acoustic features through a pre-trained voice emotion classification model, and the emotion labels are outputted, wherein the acoustic features include mel-frequency cepstral coefficients (MFCC), a fundamental frequency (F0), an energy contour and a speech rate;

[0139] (2) For video data, real-time target detection is performed on video frames by YOLOv8 or DETR model to identify the anchor, product display area and key actions, the product packaging information is identified by OCR and verified by the product library to verify the authenticity of the product, the product packaging information includes the brand, model and efficacy label of the product, the video data corresponding to the video is detected to determine whether there is a fake situation, the output picture credibility score, the fake situation includes editing traces, filter transition use and special effect forgery, and the anchor's body movement is extracted by a pose estimation model, the anchor's expression is analyzed by a face recognition model to determine whether there is exaggerated performance, and an action expression label is output;

[0140] (3) For text data, the data cleaning removes advertising law prohibited words, repeated content and irrelevant symbols, and retains the core propaganda language, the pre-trained language model is used to perform word segmentation, part-of-speech tagging and named entity recognition on the cleaned text data, and the key entities and semantic relationships are extracted, wherein the key entities include product name, efficacy promotion and sales data, the corresponding key entity and semantic relationship instances include product-efficacy-whitening, the sentiment classification model based on BERT fine-tuning is used to determine the sentiment tendency of the text data, and the intention classification model is used to identify whether there is an induced language, and a sentiment intention label is generated.

[0141] This module converts unstructured multi-modal data into structured features that can be processed by the model, while filtering noise interference.

[0142] Voice data processing and feature extraction includes the following operations:

[0143] (1) Speech to text (ASR): Use a Transformer-based end-to-end speech recognition model (such as Wav2Vec2.0 or Conformer) to perform real-time recognition on 1-second speech streams, outputting timestamped text (such as "00:01:23-00:01:25 anchor says: This mask has 3 times the moisturizing effect of ordinary masks");

[0144] (2) Acoustic feature extraction: Extract Mel-frequency cepstral coefficients (MFCC), fundamental frequency (F0), energy contour (Energy Contour), speech rate (words per minute), etc. Features of the voice are used to analyze the anchor's tone (such as excited, flat), emotion (such as excited, anxious);

[0145] (3) Sentiment classification: Use a pre-trained speech sentiment classification model (such as a CNN-LSTM-based model) to classify the sentiment labels of the speech segment (such as "exaggerated", "induced", "neutral"), which can help determine the false propaganda tendency.

[0146] Video data processing and feature extraction, including the following operations:

[0147] (1) Target detection and segmentation: Real-time target detection on video frames using YOLOv8 or DETR model, identifying anchors, product display areas (such as products placed on the table, experimental props), and key actions (such as "picking up product display" "applying skin care products");

[0148] (2) Product information extraction: Identifying brand, model, and efficacy annotations (such as "whitening and fading spots" "24-hour moisturizing") on product packaging through OCR, and verifying product authenticity in combination with a product library (previously interfaced with e-commerce platform product database);

[0149] (3) Picture quality analysis: Detecting whether the video has editing traces (such as frame skipping, repeated segments), excessive use of filters (such as skin smoothing that makes skin imperfections invisible), and special effects (such as virtual like number animations), and outputting picture credibility scores;

[0150] (4) Action and expression analysis: Extracting anchor limb movements (such as hand gesture amplitude and frequency) through a pose estimation model (such as OpenPose), analyzing expressions (such as excessive smiling and eye avoidance) through a facial expression recognition model (such as a ResNet-50-based expression classification model), and determining whether there are exaggerated performances.

[0151] Text data processing and feature extraction includes the following operations:

[0152] (1) Text cleaning: Removing advertising banned words (such as "best" "first" "absolute"), repetitive content, and irrelevant symbols (such as emoticons and links), and retaining core promotional language;

[0153] (2) Semantic understanding: Performing word segmentation, part-of-speech tagging, and named entity recognition (NER) on the cleaned text through a pre-trained language model (such as BERT-base-Chinese), extracting key entities (such as product name, efficacy claims, and sales data) and semantic relationships (such as "product A → efficacy → whitening");

[0154] (3) Sentiment and intent analysis: Using a BERT fine-tuned sentiment classification model to determine the sentiment of the text (such as "exaggerated" "false promise" "neutral"), and using an intent classification model to identify whether there are inducement tactics (such as "limited-time flash sale" "only 10 left").

[0155] Multi-modal fusion is used to perform the following: Fine-grained alignment and fusion of speech features, video features, and text features through cross-modal attention mechanism, obtain multi-modal features, and perform dimensionality reduction operation on multi-modal features.

[0156] As a specific implementation of the multi-modal fusion module, the module is used to perform the following operations:

[0157] (1) Text-voice alignment: align the timestamped text with the acoustic features of the original voice;

[0158] (2) Video-text alignment: align the target object detection box extracted from the video data with the target object position mentioned in the text, analyze whether the actual effect in the video screen contradicts the text propaganda;

[0159] (3) Voice-video alignment: align the emotion label based on voice data analysis with the action body based on video data analysis, and identify whether there is a transition performance type of propaganda;

[0160] (4) Fusion feature dimension reduction: generate a fused feature based on text-voice alignment, video-text alignment, and voice-video alignment, and the dimension of the fused feature is the sum of the dimensions of the text feature, the voice feature, and the video feature. The fused feature is reduced to a predetermined dimension through a fully connected layer.

[0161] The modal detection is used to perform the following: based on a Transformer model, a large model is constructed, and a multi-modal feature is input. After training, the large model is used for false propaganda detection, and the detection result including whether it is false propaganda, the violation type, and the confidence is output.

[0162] The large model based on the Transformer model in this embodiment includes three single-modal encoders and one cross-modal fusion encoder. The three single-modal encoders are a text encoder, a voice encoder, and a video encoder. The text encoder is used for self-attention encoding of the text feature. The voice encoder is used for self-attention encoding of the voice feature. The video encoder is used for self-attention encoding of the video feature. The cross-modal fusion encoder takes the output of the three single-modal encoders as input, learns the inter-modal correlation through a multi-head attention mechanism, and performs false propaganda detection, and outputs the detection result.

[0163] During training of the large model, the live broadcast data set is labeled, and the training is performed through loss function minimization. The live broadcast data set includes normal live broadcast and false propaganda live broadcast. The loss function adopts a multi-task learning loss, including a false propaganda classification loss, a violation type classification loss, and a multi-modal alignment loss. The false propaganda classification loss and the violation type classification loss are cross-entropy losses, and the multi-modal alignment loss is a contrast loss.

[0164] After training, the large model is quantized and knowledge distilled, and then deployed for use.

[0165] In this embodiment, the model architecture is a Transformer-based multi-modal encoder, which includes three single-modal encoders (text encoder, voice encoder, and video encoder) and one cross-modal fusion encoder.

[0166] Single-modal encoder: self-attention encoding of text, speech, and video features to capture intra-modal semantic associations.

[0167] Cross-modal fusion encoder: learning inter-modal associations through multi-head attention mechanism (Multi-Head Attention), e.g., the association between the text "top sales" and the "background sales data screenshot" in the video.

[0168] Training data: using a labeled live streaming dataset containing 100,000 samples, with 50,000 normal live streams (no false propaganda) and 50,000 false propaganda (annotated violation types: exaggerating efficacy, fabricating sales, making false promises, and fabricating experimental effects, etc.).

[0169] Loss function: using multi-task learning loss, including:

[0170] False propaganda classification loss (cross-entropy loss, predicting whether it is false propaganda);

[0171] Violation type classification loss (cross-entropy loss, predicting the specific violation type);

[0172] Multi-modal alignment loss (contrastive loss, pulling closer related modal features and pushing away irrelevant features);

[0173] Real-time inference optimization: through model quantization (FP32→INT8), knowledge distillation (compressing large models into lightweight student models), and TensorRT acceleration, achieving inference delay ≤500ms on GPU servers, meeting real-time requirements (live streaming frame processing delay ≤200ms).

[0174] Early warning intervention is used to perform the following: according to the confidence level, a hierarchical early warning is conducted, high-risk detection results are pushed to the regulatory department in real time and trigger live streaming pause instructions, medium and low-risk detection results are recorded for review, and early warning intervention operations are recorded to form intervention records.

[0175] This module realizes the closed-loop management of "detection-early warning-disposal", improving the efficiency of supervision.

[0176] Hierarchical early warning: according to the confidence level of the detection model (e.g., 0-1 points), the violation risk is divided into three levels:

[0177] High risk (confidence ≥0.9): immediately trigger early warning, push a pop-up window notification (including violation timestamp, type, and live streaming segment link) to the regulatory department, and automatically pause live streaming (requires platform cooperation);

[0178] Medium risk (0.7≤confidence<0.9): push early warning information to the regulatory department for manual review;

[0179] Low risk (confidence < 0.7): record to the log for subsequent analysis to optimize the model.

[0180] Cross-departmental collaboration: through the permission management function of the blockchain, different query permissions are allocated to market supervision departments, public security departments, and consumer associations, ensuring data security while realizing collaborative law enforcement (such as market supervision departments retrieving commodity qualification data, and public security departments verifying criminal responsibility for false propaganda).

[0181] Evidence storage and traceability is used to perform the following: calculate the hash value of the original multi-modal data, detection results, and intervention records respectively and encapsulate the metadata, write into the blockchain through the PBFT consensus mechanism of the alliance chain, form an unalterable full-link evidence, and support bidirectional traceability of supervision and consumers.

[0182] As a specific implementation of the evidence storage and traceability module, the module is used to perform the following operations:

[0183] (1) Data hash generation: calculate the hash value of the real-time collected original multi-modal data and detection results respectively to generate data hash;

[0184] (2) Data encapsulation: add metadata to each data hash, wherein the metadata includes basic information, timestamp, and association identifier, the basic information includes live room number, anchor ID, start / end time, and platform ID, the timestamp includes original multi-modal data collection time and data hash generation time, and the association identifier is the association relationship of the multi-modal data;

[0185] (3) On-chain storage: package the data hash and metadata into a block, and write it into the blockchain through the PBFT consensus mechanism of the alliance chain;

[0186] (4) Evidence storage verification: each block contains the hash value of the previous block, forming a chain structure, and the supervision department queries the matching of the hash value of any block with the original data through the blockchain browser, and verifies the matching by recalculating the hash value when calculating the matching.

[0187] As a specific implementation of the evidence storage and traceability, it can realize supervision end traceability and consumer end traceability.

[0188] When the supervision end traceability, the supervisor queries the complete evidence chain of the live broadcast through the blockchain node, from the start of the live broadcast to the time period when the violation occurs, marks the data hash and detection results at each time point, obtains the original voice data, video data, and text data from the evidence storage system through the data hash, and verifies the consistency of the propaganda content and the actual commodity.

[0189] When the consumer traces the source, a blockchain storage certificate two-dimensional code is embedded in the live page, the consumer scans the storage certificate two-dimensional code and jumps to the trace page, and basic information of the host, a risk prompt, a key evidence chain and historical data comparison are displayed, video clips of a predetermined time before and after a violation time point in the key evidence chain and corresponding voice text and detection results are selected, and downloading or notarization is supported, when the historical data comparison, an API interface of historical live data stored by the blockchain is opened, the consumer searches past live records through a product name and a host ID, analyzes whether there is a mode of multiple false propaganda of the same product or repeated use of similar rhetoric by the same host, and generates a risk report.

[0190] The live e-commerce multi-modal false propaganda content detection and trace method and system provided by the application are described in detail, specific examples are applied to describe the principles and implementation modes of the application, the above example is only used to help understand the method and core idea of the application; meanwhile, for those skilled in the art, according to the idea of the application, the specific implementation mode and application range will be changed, and the above description should not be understood as a limitation of the application.

Claims

1. A method for detecting and tracing the source of multimodal false advertising content in live-streaming e-commerce, characterized in that, Includes the following steps: Data Acquisition: Multimodal data during the live broadcast is collected in real time through the live broadcast platform interface, and the multimodal data is timestamped and synchronized. The multimodal data is unstructured data, including text data, voice data and video data. Feature extraction: Perform speech analysis, feature extraction, and sentiment classification on speech data, and output timestamped text, acoustic features, and sentiment tags as speech features. Perform object detection, product information verification, image quality analysis, and action and expression analysis on video data, and output object detection boxes, product authenticity verification results, image credibility scores, and action and expression tags as video features. Perform semantic understanding and sentiment intent analysis on text data, and output entity-semantic relationships and sentiment intent tags as text features. Multimodal fusion: Fine-grained alignment and fusion of speech features, video features, and text features are performed through a cross-modal attention mechanism to obtain multimodal features, and then dimensionality reduction is performed on the multimodal features; Modal detection: A large model is built based on the Transformer model. Multimodal features are used as input. The trained large model is used to detect false advertising and outputs the detection results, including whether it is false advertising, the type of violation, and the confidence level. Early warning and intervention: Early warnings are issued in tiers based on confidence level. High-risk detection results are pushed to regulatory authorities in real time and trigger a live broadcast suspension order. Medium and low-risk detection results are recorded for review, and early warning and intervention operations are recorded to form an intervention record. Evidence preservation and traceability: The original multimodal data, test results and intervention records are hashed and encapsulated with metadata. They are then written into the blockchain through the PBFT consensus mechanism of the consortium blockchain to form tamper-proof end-to-end evidence, supporting two-way traceability for regulators and consumers.

2. The method for detecting and tracing the source of multimodal false advertising content in live-streaming e-commerce according to claim 1, characterized in that, During data collection, audio, video, and text data are collected synchronously through the inference API or streaming interface provided by the live streaming platform. Audio data includes the audio stream of the anchor's real-time narration, video data includes the live video stream, and text data includes real-time text appearing in the live video, as well as bullet screen / comment data obtained from the live streaming platform through the WebSocket interface, which is also recognized by OCR tools. A unified timestamp is added to all multimodal data, and discretely collected multimodal data is cached and aligned using a sliding window.

3. The method for detecting and tracing the source of multimodal false advertising content in live-streaming e-commerce according to claim 1, characterized in that, Feature extraction includes the following operations: For speech data, real-time recognition is performed using an end-to-end speech recognition model based on Transformer, outputting timestamped text and extracting acoustic features of the speech. Based on the extracted acoustic features, sentiment labels are classified using a pre-trained speech sentiment classification model, and sentiment labels are output. The acoustic features include Mel-frequency cepstral coefficients (MFCC), fundamental frequency (F0), energy contour, and speech rate. For video data, real-time target detection is performed on video frames using YOLOv8 or DETR models to identify the anchor, product display area, and key actions. Product packaging information is identified using OCR and verified for authenticity using a product database. Product packaging information includes the brand, model, and efficacy label of the product. The system detects whether the video data is counterfeited and outputs a credibility score. Counterfeiting includes editing marks, excessive use of filters, and fake special effects. The anchor's body movements are extracted using a pose estimation model, and the anchor's facial expressions are analyzed using a facial recognition model to determine whether there is exaggerated performance behavior and output action and expression labels. For text data, data cleaning removes prohibited words, duplicate content, and irrelevant symbols, retaining core promotional phrases. A pre-trained language model is used to segment, tag, and name entity recognition of the cleaned text data, extracting key entities and semantic relationships. Key entities include product names, efficacy claims, and sales data. Examples of key entities and semantic relationships include product-efficacy-whitening. A sentiment classification model fine-tuned based on BERT is used to determine the sentiment tendency of the text data, and an intent classification model is used to identify the presence of misleading phrases, generating sentiment intent labels.

4. The method for detecting and tracing the source of multimodal false advertising content in live-streaming e-commerce according to claim 1, characterized in that, Multimodal fusion includes the following operations: Text-to-speech alignment: Aligning timestamped text with the acoustic features of the original speech; Video-Text Alignment: Align the target object detection boxes extracted from video data with the target object positions mentioned in the text, and analyze whether the actual effect in the video footage contradicts the text promotion; Voice-video alignment: Aligning emotion tags based on voice data analysis with body language based on video data analysis to identify the presence of overly performative or theatrical advertising; Feature dimensionality reduction: Based on text-speech alignment, video-text alignment, and speech-video alignment, fused features are generated. The dimensionality of the fused features is the sum of the text feature dimension, speech feature dimension, and video feature dimension. The fused features are then reduced to a predetermined dimension through a fully connected layer.

5. The method for detecting and tracing the source of multimodal false advertising content in live-streaming e-commerce according to claim 1, characterized in that, The large model built on the Transformer model includes three unimodal encoders and one cross-modal fusion encoder. The three unimodal encoders are a text encoder, a speech encoder, and a video encoder. The text encoder is used to perform self-attention encoding on text features, the speech encoder is used to perform self-attention encoding on speech features, and the video encoder is used to perform self-attention encoding on video features. The cross-modal fusion encoder takes the outputs of the three unimodal encoders as input and is used to learn the intermodal correlations through a multi-head attention mechanism and perform false advertising detection, outputting the detection results. When training the large model, it is trained based on the labeled live streaming dataset and by minimizing the loss function. The live streaming dataset includes normal live streaming and false advertising live streaming. The loss function adopts multi-task learning loss, including false advertising classification loss, violation type classification loss and multimodal alignment loss. False advertising classification loss and violation type classification loss are cross-entropy loss, and multimodal alignment loss is contrast loss. The trained large model is quantized and knowledge distilled before deployment.

6. The method for detecting and tracing the source of multimodal false advertising content in live-stream e-commerce according to claim 1, characterized in that, Evidence preservation and traceability include the following operations: Data hash generation: Calculate hash values ​​for the raw multimodal data and detection results acquired in real time to generate data hashes; Data encapsulation: Add metadata to each data hash, which includes basic information, timestamp, and association identifier. The basic information includes the live room number, the streamer ID, the start / end time, and the platform ID. The timestamp includes the original multimodal data collection time and the data hash generation time. The association identifier is the association relationship of the multimodal data. On-chain storage: Pack data hashes and metadata into blocks and write them to the blockchain through the PBFT consensus mechanism of the consortium blockchain; Evidence verification: Each block contains the hash value of the previous block, forming a chain structure. Regulatory authorities can use a blockchain explorer to query the matching of the hash value of any block with the original data. When calculating the matching, the hash value is recalculated for verification.

7. The method for detecting and tracing the source of multimodal false advertising content in live-streaming e-commerce according to claim 6, characterized in that, Evidence preservation and traceability include traceability at the regulatory end and traceability at the consumer end; When tracing the source of the problem, regulators use blockchain nodes to query the complete evidence storage chain of the live stream. From the start of the live stream to the time period when the violation occurred, they mark the data hash and detection results at each time point. They then use the data hash to obtain the original audio data, video data, and text data from the evidence storage system to verify the consistency between the promotional content and the actual product. When consumers trace the source of the product, a blockchain-based evidence QR code is embedded in the live stream page. After scanning the QR code, consumers are redirected to the traceability page, which displays the host's basic information, risk warnings, key evidence chains, and historical data comparisons. The key evidence chain selects video clips from before and after the violation time point, along with corresponding audio and text recordings and detection results, which can be downloaded or notarized. For historical data comparison, the blockchain-stored historical live stream data has an open API interface, allowing consumers to search for past live stream records by product name and host ID. This analyzes whether there are multiple instances of false advertising for the same product or repeated use of similar language by the same host, and generates a risk report.

8. A system for detecting and tracing multimodal false advertising content in live-streaming e-commerce, characterized in that, The system is used to detect and trace false advertising content in live-stream e-commerce using a method as described in any one of claims 1-7. The system includes a data acquisition module, a feature extraction module, a multimodal fusion module, a modality detection module, an early warning and intervention module, and an evidence storage and traceability module. The data acquisition module is used to perform the following: collect multimodal data in real time during the live broadcast through the live broadcast platform interface, and synchronize the timestamp of the multimodal data. The multimodal data is unstructured data, including text data, voice data and video data. Feature extraction is used to perform the following: speech analysis, feature extraction and sentiment classification on speech data, outputting time-stamped text, acoustic features and sentiment tags as speech features; object detection, product information verification, picture quality analysis and action and expression analysis on video data, outputting object detection boxes, product authenticity verification results, picture credibility score and action and expression tags as video features; semantic understanding and sentiment intent analysis on text data, outputting entity-semantic relationship and sentiment intent tags as text features. Multimodal fusion is used to perform the following: fine-grained alignment and fusion of speech features, video features and text features through a cross-modal attention mechanism to obtain multimodal features, and then dimensionality reduction operation on the multimodal features; Modality detection is used to perform the following: a large model is built based on the Transformer model, multimodal features are taken as input, and false advertising is detected through the trained large model. The output includes whether false advertising is used, the type of violation, and the confidence level. The early warning intervention is used to perform the following: tiered early warnings are issued based on confidence levels; high-risk detection results are pushed to regulatory authorities in real time and trigger a live broadcast pause command; medium- and low-risk detection results are recorded for review, and the early warning intervention operation is recorded to form an intervention record; Evidence preservation and traceability are used to perform the following: calculate hash values ​​for the original multimodal data, detection results, and intervention records, encapsulate metadata, and write them to the blockchain through the PBFT consensus mechanism of the consortium blockchain to form tamper-proof end-to-end evidence, supporting two-way traceability for regulators and consumers.

Images