Multimodal recognition e-commerce live streaming teaching system
The e-commerce live-streaming teaching system, which utilizes multimodal recognition and integrates a multi-layered architecture and innovative modules, solves the problems of multi-dimensional evaluation and compliance lag in existing systems. It achieves high-precision behavior detection and dynamic compliance adaptation, improving teaching quality and efficiency, while also supporting barrier-free use by special groups.
Patent Information
- Application Number
- CN202611268035.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-20
- Publication Date
- 2026-11-17
AI Technical Summary
Existing e-commerce live streaming teaching systems lack multimodal data fusion and analysis capabilities, making it impossible to achieve comprehensive evaluation of multi-dimensional information such as visual, auditory, and textual data. Furthermore, compliance detection is lagging, making it difficult to provide real-time warnings of violations and failing to adapt to the differentiated compliance rules of different countries or regions.
A multimodal recognition e-commerce live streaming teaching system was designed, integrating a user interaction layer, a core business logic layer, a multi-threaded engine layer, a device acquisition layer, and a data and knowledge base layer. It adopts modules such as an eye-tracking engine, a behavior compliance judge, and a speech performance evaluator to achieve comprehensive intelligent monitoring and guidance. Combined with a close-up display anti-false alarm mutual exclusion algorithm, a two-stage triggered gesture compliance filtering algorithm, a closed-loop bullet screen interaction evaluation system, and a dynamic compliance knowledge base, it supports multi-threaded parallel processing and dynamic compliance knowledge updates.
It achieves high-precision behavioral compliance detection, dynamic compliance knowledge adaptation, immersive interactive training experience, provides a comprehensive teaching evaluation system, and supports barrier-free adaptation, thereby improving teaching efficiency and compliance.
Smart Images

Figure CN123118316A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of e-commerce live streaming, and in particular to a multimodal recognition e-commerce live streaming teaching system. Background Technology
[0002] With the booming development of e-commerce, live streaming has become an important marketing tool, placing extremely high demands on the professional competence and compliance awareness of live streamers. Currently, e-commerce live streaming training mainly relies on manual guidance. Instructors need to simultaneously monitor trainees' language expression, body language, product display, and platform compliance requirements, resulting in a heavy workload and difficulty in achieving refined assessment. While existing technologies include single-mode speech recognition or motion capture systems, they often lack the ability to fuse and analyze multimodal data, failing to achieve comprehensive evaluation of visual, auditory, and textual information. Furthermore, existing teaching systems are typically lagging in detecting live streaming compliance, making it difficult to provide real-time warnings for inappropriate gestures, vulgar actions, or prohibited language during practical training. This leads to trainees developing poor live streaming habits, and there is a lack of dynamic adaptation mechanisms for the differentiated compliance rules of different countries or regions. Therefore, there is an urgent need for an e-commerce live streaming teaching system that integrates multimodal recognition technology and possesses real-time compliance warnings and intelligent teaching assessment functions. Summary of the Invention
[0003] The present invention aims to at least partially solve one of the technical problems in the related art.
[0004] Therefore, the purpose of this invention is to propose a multimodal recognition e-commerce live streaming teaching system, which integrates a multi-layer architecture including user interaction, core business logic, multi-threaded engine, device acquisition and data knowledge base, and realizes comprehensive and intelligent monitoring and guidance of the live streaming training process.
[0005] To achieve the above objectives, this invention proposes a multimodal recognition e-commerce live streaming teaching system, comprising a user interaction layer, a core business logic layer, a multi-threaded engine layer, a device acquisition layer, and a data and knowledge base layer. The user interaction layer includes a student interface, a teacher interface, and an administrator backend. The core business logic layer includes a graph analyzer, a behavior compliance determiner, a speech performance evaluator, a bullet screen event generator, a benchmarking matching engine, and an eye-tracking engine. The multi-threaded engine layer includes a CV Engine, an ASR consumer, a TTS queue, and a compliance updater. The device acquisition layer includes camera acquisition and microphone acquisition. The data and knowledge base layer includes an SQLite compliance library, a vector database, and a training record library. The system integrates four innovative modules: a close-up display false alarm prevention mutual exclusion algorithm, a two-stage triggered gesture compliance filtering algorithm, a closed-loop bullet screen interaction evaluation system, and a dynamic compliance knowledge base based on RAG.
[0006] In addition, the e-commerce live-streaming teaching system based on multimodal recognition proposed above according to the present invention may also have the following additional technical features:
[0007] Specifically, the eye-tracking engine detects facial occlusion, inappropriate gestures, vulgar actions, and dangerous product displays. Facial occlusion specifically refers to hands or products obscuring the facial contours for extended periods. Inappropriate gestures refer to actions prohibited by the platform. Vulgar actions refer to the exposure of sensitive areas and inappropriate postures. Dangerous product displays refer to the display of prohibited items. The behavior compliance judge's detection method is as follows: 1. The TFLite model outputs the gesture confidence score. When the confidence score is greater than 0.75, it is pushed into a 15-frame double-ended queue without triggering an alarm. 2. The density of inappropriate frames in the queue is calculated in real time. Only when more than 80% of the past 15 frames contain the same inappropriate gesture is a true violation confirmed. After triggering an alarm, a 3000ms cooldown period is entered to prevent UI flickering.
[0008] Specifically, the warning levels for facial obstruction, inappropriate gestures, vulgar actions, and display of dangerous goods are set, including yellow alerts and red blocking. The warning level for facial obstruction is a yellow alert. The warning level for inappropriate gestures, vulgar actions, and display of dangerous goods is a red blocking warning.
[0009] Specifically, the bullet comment event generator includes script mounting and large model generation. The script mounting is based on a preset bullet comment library in script_danmaku.json and randomly extracted by QTimer. The large model generation is achieved by Qwen-1.8B generating argument bullet comments in real time based on ASR text.
[0010] Specifically, the action similarity calculation of the benchmark matching engine is based on the dynamic time warping algorithm of the skeletal point sequence. The calculation process includes skeletal point extraction, sequence alignment, similarity calculation, and feedback generation. Specifically, skeletal point extraction involves extracting the coordinates of 18 key skeletal points from the video stream. Sequence alignment involves aligning the action sequences of the student and the demonstration using the dynamic time warping algorithm based on the skeletal point sequence. The similarity calculation involves joint angle similarity, action rhythm similarity, and spatial position similarity. The feedback generation involves displaying the overlap percentage in real time and providing specific adjustment suggestions.
[0011] Specifically, the behavior compliance determiner includes dual-level hierarchical detection, offline compliance package support, and RAG dynamic compliance updates. The dual-level hierarchical detection includes domestic training detection and cross-border training detection. The offline compliance package supports downloading offline compliance packages for the target country, allowing training to be conducted even when the network is offline. The RAG dynamic compliance update uses a combination of SQLite and keyword matching. The background thread pulls the latest list of prohibited words from the cloud every day and writes it to the local SQLite database. During training, Aho-Corasick multi-mode matching is used.
[0012] Specifically, the speech performance evaluator includes low-latency real-time transcription, speech compliance and sensitive word detection, and performance and emotion evaluation. The low-latency real-time transcription uses WebSocket to establish a long connection and transmits audio in real time in 16000Hz PCM format, achieving millisecond-level text display. The speech compliance and sensitive word detection types include prohibited word detection, semantic violation detection, and extreme word detection. The specific implementation methods correspond to Aho-Corasick multi-pattern matching, RAG vector similarity retrieval, and extreme word detection. The evaluation dimensions of the performance and emotion evaluation include speech rate, volume, emotional appeal, and speech fluency.
[0013] Specifically, the system also includes a teacher-side evaluation management module, which includes automatic objective data statistics, human-computer collaborative scoring, and batch class management. The statistical items for automatic objective data statistics include training duration, number of violations, diagram completion rate, benchmark matching degree, interaction success rate, and fluency of speech. The scoring mechanism for human-computer collaborative scoring is based on the final score, which is obtained by adding the base score and the bonus score. The batch class management includes creating classes, setting topics, configuring weights, and exporting reports.
[0014] Specifically, the system also includes a student-side training assistance module, which includes real-time screen guidance, prompt word linkage, and automatic generation of training reports. The real-time screen guidance has three forms: slight prompts, moderate prompts, and strong intervention. The prompt word linkage functions include script import, font scaling, automatic scrolling, and position adjustment. The automatically generated training report includes basic information, a five-dimensional radar chart, deduction details, and improvement suggestions.
[0015] Specifically, the system also includes an accessibility adaptation module, which consists of a visual impairment mode and a hearing impairment mode. The visual impairment mode uses screen reader software to automatically identify and broadcast all interface elements, while the hearing impairment mode uses different colored lights to flash to provide warnings for different content.
[0016] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:
[0017] 1. High-precision behavior compliance detection: By combining the eye-tracking engine with the behavior compliance judge, and using dual-end queue caching and violation frame density calculation, the false alarm problem caused by jitter during live streaming is effectively solved. Combined with the hierarchical early warning mechanism, it realizes accurate identification and differentiated intervention for violations.
[0018] 2. Dynamic compliance knowledge adaptation: Based on the RAG dynamic compliance knowledge base and offline compliance package support, the system can adapt to the training needs in offline environments and can synchronize the latest platform compliance rules in real time, solving the problem of lagging compliance rule updates in traditional systems.
[0019] 3. Immersive interactive training experience: The bullet screen event generator combines preset scripts with large model generation technology to simulate the complex interactive scenarios of a real live broadcast room, improving trainees' ability to adapt to the situation; the benchmark matching engine uses the dynamic time warping algorithm of skeletal point sequence to achieve precise quantitative evaluation and guidance of action details.
[0020] 4. Comprehensive teaching evaluation system: The speech performance evaluation tool realizes multi-dimensional evaluation from compliance to performance. Combined with the teacher evaluation management module and the student training assistance module, it realizes objective statistics of teaching data, intelligent scoring and personalized feedback, which significantly improves teaching efficiency and quality.
[0021] 5. Excellent accessibility support: By adapting the visually impaired mode and the hearing impaired mode, the usage rights of special groups are guaranteed, and the universality and social value of the system are enhanced.
[0022] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0023] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:
[0024] Figure 1 This is a schematic diagram of the overall e-commerce live streaming teaching system based on multimodal recognition of the present invention;
[0025] Figure 2 This is the state machine transition diagram of the e-commerce live streaming teaching system for multimodal recognition in this invention;
[0026] Figure 3 This is a flowchart of the two-stage triggering algorithm in the e-commerce live streaming teaching system for multimodal recognition of the present invention;
[0027] Figure 4 This is a flowchart illustrating the closed-loop evaluation process of bullet screen interaction in the e-commerce live streaming teaching system based on multimodal recognition, as described in this invention.
[0028] As shown in the figure: 1. User interaction layer; 2. Core business logic layer; 3. Multi-threaded engine layer; 4. Device acquisition layer; 5. Data and knowledge base layer. Detailed Implementation
[0029] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the invention, and should not be construed as limiting the invention. Rather, embodiments of the invention include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.
[0030] The following description, in conjunction with the accompanying drawings, describes a multimodal recognition e-commerce live-streaming teaching system according to an embodiment of the present invention.
[0031] like Figure 1 As shown in the figure, this embodiment provides a multimodal recognition e-commerce live streaming teaching system. The system adopts a layered architecture design, which consists of a user interaction layer 1, a core business logic layer 2, a multi-threaded engine layer 3, a device acquisition layer 4, and a data and knowledge base layer 5, from top to bottom. This layered design effectively decouples interactive display, business logic processing, and underlying data acquisition, which not only facilitates independent maintenance and upgrades of each layer, but also provides a solid physical foundation and logical boundary for the system to handle high concurrency and multimodal data.
[0032] The user interaction layer 1 includes a student interface, a teacher interface, and an administrator backend.
[0033] Specifically, User Interaction Layer 1 is the window for human-computer interaction, responsible for receiving user operation commands and displaying the processing results. The student and teacher interfaces can run on desktop computers, laptops, tablets, or smartphones, presenting live stream footage, training prompts, and grading reports through a graphical user interface. The administrator backend typically runs on the server, allowing system administrators to perform maintenance operations such as user permission allocation and system parameter configuration. The student interface receives and displays real-time audio and video data uploaded by students during live training sessions, while the teacher interface allows teachers to view class training status, intervene in real-time, and manage evaluations. The administrator backend is responsible for maintaining basic data and monitoring the system.
[0034] The core business logic layer 2 includes a graph analyzer, a behavior compliance judge, a speech performance evaluator, a bullet screen event generator, a benchmarking matching engine, and an eye tracking engine.
[0035] This module is responsible for handling complex business rules and algorithmic logic. The layout analyzer analyzes whether the layout of the live stream conforms to aesthetic or marketing standards; the behavior compliance judge works in conjunction with the eye-tracking engine to monitor the anchor's body movements, eye direction, and violations in real time; the speech performance evaluator transcribes and evaluates the audio content in multiple dimensions; the bullet screen event generator simulates a real live stream interaction environment; and the benchmarking matching engine compares the trainee's actions with standard demonstrations. These modules work together to support the core functions of e-commerce live streaming teaching.
[0036] The multi-threaded engine layer 3 includes a CV Engine, an ASR consumer, a TTS queue, and a compliance updater.
[0037] Considering the extremely high real-time requirements of live training, this layer adopts a multi-threaded parallel processing mechanism. The CV Engine runs independently on the vision processing thread, responsible for calling GPU resources to perform frame extraction and feature extraction from the video stream; the ASR consumer runs on the speech processing thread, pulling the audio stream in real time and performing recognition; the TTS queue manages the scheduling of text-to-speech tasks; and the compliance updater executes compliance library update tasks periodically in a background thread.
[0038] This multi-threaded design avoids a single task blocking the entire system process, ensuring that multimodal data such as video, audio, and text can flow synchronously and efficiently, providing millisecond-level response capabilities for upper-layer business logic.
[0039] The device acquisition layer 4 includes camera acquisition and microphone acquisition.
[0040] This layer serves as the system's data entry point, responsible for digitizing physical signals. The camera acquisition module uses a driver to call a USB camera, professional camcorder, or mobile phone's built-in camera to acquire video stream data at a set frame rate and resolution. The microphone acquisition module uses a sound card or audio driver to acquire ambient sound and the broadcaster's voice, converting them into digital audio signals. Device acquisition layer 4 encapsulates the acquired raw audio and video data into a standard format and passes it to the multi-threaded engine layer 3 for processing via the internal bus or shared memory.
[0041] The data and knowledge base layer 5 includes an SQLite compliance library, a vector database, and a training record library.
[0042] This layer serves as the system's data persistence and knowledge support center. The SQLite compliance database stores structured information such as prohibited word lists and violation action feature data, supporting fast local queries. The vector database stores embedded representations of spoken text and action feature vectors, supporting semantic retrieval and similarity matching. The training record database stores students' training process data, scoring results, and historical trajectories, providing a data foundation for subsequent data analysis and teaching evaluation. By separating the knowledge base from business logic, the system achieves decoupling of data and logic, facilitating independent knowledge updates and reuse.
[0043] The system integrates four innovative modules: a close-up display anti-false alarm mutual exclusion algorithm, a two-stage triggered gesture compliance filtering algorithm, a closed-loop bullet screen interaction evaluation system, and a dynamic compliance knowledge base based on RAG.
[0044] These four innovative modules are not independent hardware entities, but rather a collection of functional algorithms and logic integrated into the aforementioned five-layer architecture. The close-up display false alarm prevention algorithm is primarily deployed in the core business logic layer 2. By analyzing depth information or image features transmitted from the device acquisition layer 4, it distinguishes between normal close-up product displays and illegal facial occlusion, preventing false alarms. The two-stage triggered gesture compliance filtering algorithm also runs in the core business logic layer 2. Utilizing the computing power provided by the multi-threaded engine layer 3, it performs confidence filtering and density calculation on consecutive frames, effectively filtering out false judgments caused by momentary jitter. The closed-loop bullet screen interaction evaluation system runs through the core business logic layer 2 and the user interaction layer 1. After the bullet screen event generator generates bullet screens, the system monitors the trainees' reactions and responses in real time, forming a closed loop of "generation-interaction-evaluation." The RAG-based dynamic compliance knowledge base relies on the data and knowledge base layer 5 and the multi-threaded engine layer 3. Through a compliance updater, it periodically pulls the latest rules from the cloud, updating the SQLite compliance database and vector database to ensure the timeliness of compliance detection.
[0045] By organically combining the aforementioned five-layer architecture with four innovative modules, this embodiment constructs a closed-loop system covering the entire chain, from bottom-layer data acquisition, mid-layer parallel processing, upper-layer business logic to top-layer interactive feedback. Data interaction between layers is achieved through standard interfaces, ensuring both high cohesion and low coupling within the system. Furthermore, the collaboration of multi-threading and intelligent algorithms enables high-precision, low-latency, and comprehensive monitoring and guidance of the e-commerce live-streaming training process, effectively solving the problems of single-dimensional teaching evaluation, lagging compliance testing, and poor interactivity in existing technologies.
[0046] Example 2:
[0047] Based on Example 1, this example describes in detail the collaborative workflow of the eye tracking engine and the behavior compliance judge in the core business logic layer 2, focusing on its false alarm prevention mechanism and hierarchical early warning strategy.
[0048] The eye-tracking engine detects facial occlusion, inappropriate gestures, vulgar actions, and the display of dangerous items.
[0049] Specifically, the gaze tracking engine, as a front-end perception module, is responsible for analyzing video frame data transmitted from the camera acquisition layer in real time. Facial occlusion refers to hands or products obscuring the facial contours for extended periods, which is highly likely to occur during close-up displays in live-streaming e-commerce. The system needs to distinguish between "reasonable occlusion for showcasing product details" and "malicious facial occlusion." Prohibited gestures refer to actions prohibited by the platform, such as giving the middle finger or making cutting gestures—gestures that are insulting or suggestive. Vulgar actions include exposing sensitive areas and inappropriate postures, such as revealing clothing or sexually suggestive body movements. Displaying dangerous goods refers to displaying prohibited items, such as controlled knives, flammable and explosive materials, etc. The gaze tracking engine initially identifies these behaviors using skeletal point detection and image classification algorithms, and then passes the identification results to the behavior compliance judge for secondary confirmation.
[0050] The behavior compliance determiner uses a "two-stage triggered gesture compliance filtering algorithm" to solve the problem of high false alarm rate caused by single-frame detection.
[0051] The specific implementation process of the algorithm is as follows:
[0052] Step S201, First-stage confidence screening. The system calls the lightweight TFLite model to perform real-time inference on video frames. The TFLite model has the advantages of small size and fast inference speed, making it suitable for running on edge devices or ordinary teaching terminals. The model outputs the gesture confidence score, and the system presets a confidence threshold, which is preferably 0.75 in this embodiment. When the detected gesture confidence score is greater than 0.75, the system determines that the frame contains a suspected violation gesture and pushes it into a double-ended queue of 15 frames. At this time, no alarm is immediately triggered. The advantage of this design is that, through the threshold screening in the first stage, frames that the model considers "completely impossible to violate" are filtered out, reducing the amount of subsequent calculations, while the queue caching provides a data foundation for time-series analysis. It should be understood that the confidence threshold of 0.75 is an optimal value obtained by balancing recall and precision based on a large amount of actual test data. In other embodiments, it can be adjusted to 0.70 or 0.80 according to the actual false alarm situation.
[0053] Step S202, Second Stage: Calculation of Violation Frame Density. The system calculates the density of violation frames in the queue in real time. Since the length of the double-ended queue is fixed at 15 frames, the queue always stores the detection results of the most recent 15 frames. The system only confirms a real violation when the calculated density is greater than 80%, meaning that at least 12 of the past 15 frames have been judged as the same violation gesture. This mechanism effectively filters out erroneous violation signals caused by changes in lighting, momentary hand tremors, or momentary misjudgments by the model.
[0054] Step S203, Cooling-off Period Control. When the system confirms a violation and triggers an alarm, it automatically enters a 3000ms cooling-off period.
[0055] During the cooldown period, the system suspends alarm triggering for this type of violation. This design is primarily to prevent UI flickering. In actual live streaming scenarios, students may continuously be in a violation state. Without a cooldown period, the system would trigger alarms continuously at an extremely high frequency, causing warning boxes to pop up constantly on the front-end interface, severely impacting user experience and even causing program lag. By setting a 3000ms cooldown period, the system gives students sufficient time to react and rectify the situation after each alarm, while also avoiding the waste of system resources caused by high-frequency alarms.
[0056] Regarding warning levels, the face obstruction, illegal gestures, vulgar actions, and display of dangerous goods are set with corresponding warning levels, including yellow alerts and red blocking.
[0057] This embodiment designs a differentiated early warning mechanism based on the risk level of the violation and the likelihood of false alarms.
[0058] Specifically, the warning level for facial occlusion is yellow.
[0059] In live-streamed teaching, facial obscuration is sometimes used to showcase product details, indicating a low level of malicious intent and a relatively high false alarm rate. Directly classifying facial obscuration as a serious violation and issuing a red warning could disrupt the normal teaching demonstration process, causing frustration for students. Therefore, the system sets it to a yellow warning, displaying only a warning icon or text message on the front-end interface without interrupting the live stream, reminding students to adjust their posture.
[0060] The warning level for the aforementioned illegal gestures, vulgar actions, and display of dangerous goods is red (blocking).
[0061] These three types of behavior involve the platform's bottom line or laws and regulations, posing an extremely high level of risk, but with a relatively low false alarm rate. Once detected and confirmed, the system immediately triggers the red blocking mechanism, forcibly interrupting the live stream or issuing a piercing alarm, and recording the serious violation. This tiered early warning mechanism reflects the system's balance between "strict compliance" and "teaching experience," ensuring both immediate control of high-risk behaviors and providing a certain degree of leeway for low-risk behaviors.
[0062] Example 3:
[0063] Based on Example 1, this example describes in detail the internal structure of the bullet screen event generator in the core business logic layer 2 and its bimodal generation mechanism.
[0064] The bullet screen event generator aims to simulate the interactive behavior of viewers in a real live stream, providing trainees with an immersive training environment. The generator includes two parallel working modes: script mounting and large-scale model generation.
[0065] Specifically, the script mounting mode is preset by script_danmaku.json and randomly selected by QTimer.
[0066] In this mode, the system has a pre-installed structured JSON file containing a large amount of preset bullet screen text data. Bullet screen extraction events are triggered at preset time intervals. To ensure the randomness and unpredictability of the interaction, the system randomly selects a bullet screen from the currently applicable categories and displays it through the student's interface in user interaction layer 1. The advantages of this mode are low computational resource consumption, fast response speed, and complete control over the bullet screen content, making it suitable for the initial teaching stage and helping students become familiar with basic interactive response processes.
[0067] As another optional implementation, the large model generation mode is generated in real time by Qwen-1.8B based on ASR text to generate argumentative bullet comments.
[0068] In this mode, the system utilizes natural language processing technology to intelligently generate text for bullet comments, which is then passed as contextual information to the bullet comment event generator. The system calls the Qwen-1.8B large language model deployed locally or in the cloud to encapsulate the ASR text into a preset template. This mode can generate targeted interactive content based on the trainee's real-time speech, simulating the challenging public opinion environment of a real live broadcast room, and is suitable for advanced stress training or adaptability assessment scenarios.
[0069] This embodiment achieves flexible adaptation to teaching scenarios through a dual-modal generation mechanism. The system can seamlessly switch between "script mounting" and "large model generation" modes based on the configuration parameters of the training course or the teacher's real-time instructions. For example, in the "newbie guidance course," the system defaults to script mounting mode to ensure that students master basic conversation skills in a low-interference environment; while in the "practical training course," the system automatically switches to large model generation mode, training students' psychological qualities and on-the-spot adaptability through high-intensity random questioning and challenges. This design not only expands the system's protection scope but also significantly improves the practicality and training effectiveness of e-commerce live streaming teaching.
[0070] Example 4:
[0071] Based on Example 1, this example describes in detail the action similarity calculation process of the benchmarking matching engine in the core business logic layer 2.
[0072] In e-commerce live streaming training, the standardization of body movements directly affects the live streaming effect. The benchmark matching engine aims to guide trainees to correct deviations in their movements through quantitative analysis.
[0073] The action similarity calculation of the benchmark matching engine is based on the dynamic time warping algorithm of the skeleton point sequence.
[0074] The calculation process specifically includes four steps: skeletal point extraction, sequence alignment, similarity calculation, and feedback generation.
[0075] Step S401, Skeleton Point Extraction. The system extracts the coordinates of 18 key skeleton points from the video stream transmitted from the device's acquisition layer 4.
[0076] Specifically, the matching engine invokes a lightweight pose estimation model to perform inference on each frame of video image. These 18 key skeletal points cover the main motion joints and feature parts of the human body, such as: nose tip, neck, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, left and right ankles, and left and right eyes and ears. Each skeletal point contains two-dimensional coordinate information and confidence level. The system combines the skeletal point coordinates extracted from consecutive frames in chronological order to form a skeletal point sequence of the student's movements. It should be understood that although this embodiment preferably uses 18 key points, in scenarios with limited computing resources or low accuracy requirements, only the skeletal points of the upper body can be extracted to reduce the computational load.
[0077] Step S402, Sequence Alignment: The system uses a dynamic time warping algorithm based on skeletal point sequences to align the student's and the demonstration's action sequences.
[0078] This is one of the core innovations of this embodiment. In live-stream training scenarios, it is difficult for trainees to maintain a completely identical movement rhythm to the demonstration video, often resulting in movements that are too fast, too slow, or paused in the middle. Using traditional Euclidean distance to directly calculate the distance between two sequences with inconsistent lengths or misaligned timelines leads to significant errors and fails to accurately reflect the similarity of movement patterns. The Dynamic Time Warping (DTW) algorithm constructs a cost matrix and uses dynamic programming to find an optimal path, non-linearly mapping the timeline of the student's movement sequence to the timeline of the demonstration movement sequence, thus achieving alignment. For example, when a trainee's action of demonstrating a product is half a beat slower than the demonstration, the DTW algorithm can automatically match the trainee's 10th frame with the demonstration's 8th frame, rather than forcibly requiring a one-to-one correspondence between frame numbers. This mechanism ensures the system's tolerance for inconsistent movement rhythms, making the evaluation results more objective and fair.
[0079] Step S403, Similarity Calculation. Based on sequence alignment, the system calculates joint angle similarity, movement rhythm similarity, and spatial position similarity.
[0080] Step S404: Feedback Generation. The system displays the overlap percentage in real time and provides specific adjustment suggestions.
[0081] This embodiment effectively solves the problem of motion evaluation caused by individual rhythm differences in live training by introducing a dynamic time warping algorithm and a multi-dimensional similarity calculation model. It achieves comprehensive quantitative guidance on motion details, rhythm and spatial position, and significantly improves the accuracy of motion teaching and training efficiency.
[0082] Example 5:
[0083] Based on Example 1, this example describes in detail the compliance knowledge base update mechanism and offline protection strategy of the behavior compliance determiner in the core business logic layer 2.
[0084] In e-commerce live streaming training, compliance rules are highly time-sensitive and geographically specific. Platforms in different countries or regions have significantly different definitions of prohibited words and actions, and rule bases are frequently updated. To address this technical challenge, the proposed behavior compliance determiner includes dual-rule layered detection, offline compliance package support, and RAG dynamic compliance updates.
[0085] Specifically, the dual-track tiered testing includes domestic training testing and cross-border training testing.
[0086] During the initialization phase, the system automatically loads the corresponding rule set based on the training course theme or the teacher's configuration instructions. Domestic training assessments primarily rely on domestic laws and regulations such as the Advertising Law and the Administrative Measures for Online Live Streaming Marketing, as well as the community rules of mainstream domestic platforms like Douyin and Taobao, focusing on detecting superlative words such as "first," "best," and "top-tier," and specific prohibited gestures. Cross-border training assessments are based on the laws and regulations of the target country (such as the United States and Southeast Asian countries) and the rules of platforms like Amazon and TikTok, focusing on detecting racially discriminatory language, specific religiously taboo actions, or marketing rhetoric prohibited by the target country's laws. This tiered assessment mechanism ensures that the compliance determination of training content accurately matches the target market, avoiding misjudgments or omissions caused by a "one-size-fits-all" approach, and enhancing the system's international adaptability.
[0087] To ensure system availability in offline environments, the offline compliance package supports downloading offline compliance packages for the target country, allowing training to continue even when offline.
[0088] The specific implementation process is as follows:
[0089] In step S501, with internet access, the user selects the target country or platform through the administrator backend, and the system downloads the corresponding offline compliance package from the cloud server. This compliance package is an encrypted and compressed data file containing a list of prohibited words, feature vectors of prohibited actions, and matching rules for that region.
[0090] In step S502, the system stores the offline compliance package in a specified directory on the local storage medium.
[0091] Step S503: When the training begins, the system checks the network status. If the network connection is normal, the system prioritizes using the online RAG dynamic compliance library. If the system detects a network disconnection or excessive latency, it automatically switches to offline mode and loads the local offline compliance package into memory.
[0092] In step S504, the behavior compliance judge calls the offline data in memory for real-time detection. This mechanism effectively solves the problem of unstable network in the training classroom or network restrictions in cross-border training, ensuring the continuity and robustness of the teaching process.
[0093] To address the issue of system detection lag caused by frequent updates to compliance rules, the RAG dynamic compliance update adopts a combination of SQLite and keyword matching. The background thread pulls the latest list of prohibited words from the cloud every day and writes it to the local SQLite database. During the training, Aho-Corasick multi-pattern matching was used.
[0094] The specific update and matching process is as follows:
[0095] Step S511: Background thread periodically pulls data. The system maintains an independent daemon thread in the multi-threaded engine layer 3. This thread is set to automatically wake up at 2 AM every day (when system load is low) to send an HTTPS request to the cloud compliance server and pull the latest incremental package of prohibited words. It should be understood that this time can be adjusted according to the actual server load, for example, it can be set to 3 AM every Monday.
[0096] Step S512, local SQLite write. After the daemon thread downloads the incremental package, it decrypts and verifies it. After successful verification, the thread opens the local SQLite compliance database and executes transaction operations. As a lightweight embedded relational database, SQLite does not require a separate database service process, making it very suitable for localized, high-frequency read and write operations, ensuring data storage stability and query efficiency.
[0097] Step S513: When trainees are conducting live training, the speech performance evaluator or behavior compliance judge needs to perform compliance checks on the real-time transcribed text stream.
[0098] Example 6:
[0099] Based on Example 1, this example describes in detail the internal structure of the speech performance evaluator in the core business logic layer 2 and its multi-dimensional evaluation process.
[0100] In e-commerce live streaming training, the host's language expression is a key factor affecting conversion rate. The speech performance evaluation tool aims to evaluate the trainee's speech flow in real time and quantitatively from two dimensions: compliance and performance.
[0101] The speech performance evaluator includes low-latency real-time transcription, speech compliance and sensitive word detection, and performance and emotion assessment.
[0102] While the audio is being transcribed into text, the system performs parallel operations for script compliance and sensitive word detection. The detection types for script compliance and sensitive word detection include prohibited word detection, semantic violation detection, and extreme word detection.
[0103] For prohibited word detection, the system has compiled a list of prohibited words from the SQLite compliance database. When an ASR consumer outputs a text stream, the system inputs this text stream into an automaton, which then repeatedly matches all preset prohibited words. This matching method is not only highly efficient but also accurately locates the position of prohibited words in the text, facilitating highlighting or truncation by the front-end.
[0104] The specific implementation method for detecting extreme words is called extreme word detection.
[0105] Extreme terms refer to absolute terms prohibited by the Advertising Law, such as "national level," "highest level," and "best." The system maintains a list of extreme terms and uses regular expressions or keyword matching algorithms for detection.
[0106] The evaluation dimensions for expressiveness and emotion assessment include speech rate, volume, emotional persuasiveness, and fluency.
[0107] Compliance is the bottom line for live streaming, while expressiveness is its upper limit. To quantify and evaluate participants' live streaming performance, this example constructs a multi-dimensional evaluation model.
[0108] Regarding speech rate assessment, the system counts the number of effective words per unit of time.
[0109] Regarding volume assessment, the system analyzes the mean and variance of the amplitude of the PCM audio stream.
[0110] Regarding the assessment of emotional contagiousness.
[0111] Regarding the assessment of fluency, the system calculates based on pause markers and the density of interjections in the ASR recognition results, and suggests in the training report that trainees strengthen their memorization of the script or their logical organization.
[0112] Example 7:
[0113] Based on Example 1, this example describes in detail the specific application function modules of User Interaction Layer 1, focusing on the interaction logic and data processing flow of the teacher evaluation management module, the student training assistance module and the accessibility adaptation module.
[0114] The system also includes a teacher-side evaluation management module, which includes automatic objective data statistics, human-computer collaborative scoring, and batch class management.
[0115] The teacher-side interface serves as the core interactive window for teaching management. By accessing the training record database in the data and knowledge base layer 5, it provides teachers with comprehensive teaching management support.
[0116] Specifically, the statistical items automatically collected by the objective data include training duration, number of violations, composition pass rate, benchmark matching degree, interaction success rate, and fluency of speech.
[0117] These statistics are not generated out of thin air, but are the results of real-time collection and calculation by the system during practical training. Training duration is recorded by the system timer, showing the time from when a student starts to when they end their live stream. The number of violations is automatically accumulated by the behavior compliance judge when a red blocking or yellow warning event is detected. The composition pass rate is calculated by the composition analyzer in the core business logic layer 2. The composition analyzer analyzes each frame based on aesthetic rules such as the rule of thirds and the golden ratio, calculating the percentage of time a student's image meets the composition standards. The benchmark matching degree is obtained by averaging the percentage of action similarity output by the benchmark matching engine. The interaction success rate is calculated by the ratio of the total number of bullet comments recorded by the bullet comment event generator to the number of valid student replies. The fluency of speech is directly output by the speech performance evaluator. The system displays these multi-dimensional objective data in the form of visual charts on the teacher's interface, helping teachers quickly grasp the overall training level of the class and individual differences.
[0118] The scoring mechanism for the human-machine collaborative scoring is specifically based on the final score, which is obtained by adding the base score and the bonus score.
[0119] The batch management of classes includes creating classes, setting topics, configuring weights, and exporting reports.
[0120] Teachers can create new classes and import student lists in batches through the administrator backend or the teacher interface. In the theme setting phase, teachers can select preset themes such as "Beauty Livestream" or "Digital Product Livestream," and the system will automatically load the corresponding compliance rule library and comment scripts based on the theme. The weight configuration function allows teachers to adjust the weight coefficients of each dimension in the human-computer collaborative scoring. After the training, teachers can export a PDF or Excel report containing all students' training data, scoring details, and improvement suggestions with a single click, greatly reducing the administrative workload of teaching management.
[0121] The system also includes a student-side training assistance module, which includes real-time screen guidance, interactive prompts, and automatic generation of training reports.
[0122] The student interface is the main operating platform for students to conduct live training, aiming to provide students with an immersive training experience and real-time error correction guidance.
[0123] The real-time video guidance takes three forms: mild prompts, moderate prompts, and strong intervention.
[0124] These three guidance methods correspond to different levels of violation risk and training stages. Specifically, when the system detects a minor violation or the trainee is in the beginner guidance stage, a slight prompt is used, displaying semi-transparent text at the edge of the screen without interfering with the normal live stream. When a moderate violation is detected or the trainee's action matching accuracy is below a preset threshold, a moderate prompt is used, with a prominent prompt box popping up in the center of the screen, accompanied by a short prompt sound. When a serious violation is detected or the trainee fails to correct the error for an extended period, a strong intervention is implemented. The system will forcibly pause the live stream and display an error analysis and correct demonstration video in full screen. The trainee must answer questions or re-demonstrate before continuing the training. This tiered guidance mechanism ensures both the continuity of the training and the timeliness of error correction.
[0125] The linked prompt functions include script import, font scaling, automatic scrolling, and position adjustment.
[0126] To assist learners in memorizing scripts, the student interface supports a teleprompter function. Learners can import their live stream scripts into the system beforehand, and the system uses natural language processing technology to divide the scripts into several segments. During the live stream, the teleprompter automatically scrolls to the corresponding script segment based on the speech content recognized in real time by the ASR (Automatic Speech Recognition) user, achieving "synchronization between voice and text." A font scaling function allows learners to adjust the teleprompter font size according to their viewing distance. A position adjustment function allows learners to drag the teleprompter window to any position on the screen to avoid obstructing product display areas or their faces. It should be understood that the transparency of the teleprompter window can also be adjusted to adapt to different live stream backgrounds.
[0127] The automatically generated training report includes basic information, a five-dimensional radar chart, deduction details, and improvement suggestions.
[0128] After the training, the system automatically generates a visual training report. Basic information includes the training time, theme, and duration. A five-dimensional radar chart visually displays the trainees' ability distribution across five dimensions: compliance, expressiveness, interactivity, action standardization, and composition aesthetics. The deduction details list the time, type, and corresponding deduction value for each violation. Improvement suggestions are automatically generated by the system based on the training data.
[0129] The system also includes an accessibility adaptation module, which consists of a visual impairment mode and a hearing impairment mode.
[0130] This embodiment fully considers the educational rights of special groups and eliminates the digital divide through technological means.
[0131] The visually impaired mode uses screen reader software to automatically recognize and broadcast all interface elements.
[0132] When a student activates visually impaired mode, the system adapts to the underlying controls of the student's interface.
[0133] The hearing impairment mode uses different colored lights to flash and provide warnings for different content.
[0134] For hearing-impaired trainees, traditional voice prompts or alerts are ineffective. This embodiment establishes a "sound-light" mapping mechanism. Specifically, the system invokes the LED flashlight of the terminal device or an external smart light strip, transmitting information through different flashing frequencies and color codes. For example, when the system detects a yellow alert, it controls the light to flash yellow slowly; when it detects a red blockage, it controls the light to flash red rapidly; when the training is successfully completed or the interaction is successful, it controls the light to remain constantly green. Hearing-impaired trainees can perceive the live broadcast status and violation warnings in real time by visually capturing the light signals, thereby adjusting their behavior accordingly. This multimodal interactive adaptation design reflects the system's humanistic care and expands the system's applicable user base.
[0135] This embodiment verifies the comprehensive effectiveness of the system described in this invention in terms of multimodal data fusion, real-time compliance early warning, intelligent interactive generation, and refined evaluation through a complete cross-border live streaming training process. The system not only effectively avoids compliance risks in cross-border live streaming but also significantly improves trainees' live streaming skills and professional qualities through real-time data feedback and guidance.
[0136] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0137] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0138] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. A multimodal recognition e-commerce live-streaming teaching system, characterized in that, It includes a user interaction layer (1), a core business logic layer (2), a multi-threaded engine layer (3), a device acquisition layer (4), and a data and knowledge base layer (5), among which, The user interaction layer (1) includes a student interface, a teacher interface, and an administrator backend; The core business logic layer (2) includes a graph analyzer, a behavior compliance judge, a speech performance evaluator, a bullet screen event generator, a benchmarking matching engine, and an eye tracking engine. The multi-threaded engine layer (3) includes CV Engine, ASR consumer, TTS queue and compliance updater; The device acquisition layer (4) includes camera acquisition and microphone acquisition; The data and knowledge base layer (5) includes an SQLite compliance library, a vector database, and a training record library; The system integrates four innovative modules: a close-up display anti-false alarm mutual exclusion algorithm, a two-stage triggered gesture compliance filtering algorithm, a closed-loop bullet screen interaction evaluation system, and a dynamic compliance knowledge base based on RAG.
2. The e-commerce live-streaming teaching system with multimodal recognition according to claim 1, characterized in that, The eye-tracking engine detects facial occlusion, inappropriate gestures, vulgar movements, and the display of dangerous items. The specific content of the face obstruction is that the face outline is covered by hands or goods for a long time. The specific content of the prohibited gestures is a prohibited action on the platform; The vulgar actions specifically involve the exposure of sensitive areas and inappropriate postures; The dangerous goods display specifically refers to the display of prohibited items. The detection method of the behavior compliance determiner is as follows: Step 1: The TFLite model outputs the gesture confidence score. When the confidence score is greater than 0.75, it is pushed into a 15-frame double-ended queue without triggering an alarm. Step 2: Calculate the density of violation frames in the queue in real time. Only when more than 80% of the past 15 frames contain the same violation gesture is a real violation confirmed. The system enters a 3000ms cooldown period after triggering the alarm to prevent UI flickering.
3. The e-commerce live-streaming teaching system with multimodal recognition according to claim 2, characterized in that, The system sets corresponding warning levels for facial obstruction, inappropriate gestures, vulgar actions, and the display of dangerous materials. These warning levels include a yellow alert and a red blocking warning. The warning level for facial obstruction is a yellow alert. The warning level for inappropriate gestures, vulgar actions, and the display of dangerous materials is a red blocking warning.
4. The e-commerce live-streaming teaching system with multimodal recognition according to claim 1, characterized in that, The bullet screen event generator includes script mounting and large model generation, wherein... The script is mounted using a preset bullet comment library in script_danmaku.json, which is randomly selected by QTimer. The large model is generated by Qwen-1.8B in real time from ASR text to generate argumentative bullet comments.
5. The e-commerce live-streaming teaching system with multimodal recognition according to claim 1, characterized in that, The action similarity calculation of the benchmarking matching engine is based on a dynamic time warping algorithm for skeletal point sequences. The calculation process includes skeletal point extraction, sequence alignment, similarity calculation, and feedback generation. The bone point extraction specifically involves extracting the coordinates of 18 key bone points from the video stream; The sequence alignment specifically involves using a dynamic time warping algorithm based on skeletal point sequences to align the student's and the demonstration's action sequences; The similarity calculation specifically includes joint angle similarity, movement rhythm similarity, and spatial position similarity; The feedback generation specifically involves displaying the overlap percentage in real time and providing specific adjustment suggestions.
6. The e-commerce live-streaming teaching system with multimodal recognition according to claim 1, characterized in that, The behavioral compliance assessor includes dual-standard layered detection, offline compliance package support, and RAG dynamic compliance updates. The dual-track tiered testing includes domestic training testing and cross-border training testing. The offline compliance package supports downloading offline compliance packages for the target country, allowing training to be conducted even when the internet is disconnected. The RAG dynamic compliance update adopts a combination of SQLite and keyword matching. The background thread pulls the latest list of prohibited words from the cloud every day and writes it into the local SQLite database. Aho-Corasick multi-pattern matching is used in the training.
7. The e-commerce live-streaming teaching system with multimodal recognition according to claim 1, characterized in that, The speech performance evaluator includes low-latency real-time transcription, speech compliance and sensitive word detection, and performance and emotion evaluation. The low-latency real-time transcription uses WebSocket to establish a long connection and transmits audio in real time in 16000Hz PCM format, achieving millisecond-level text display. The types of detection for the compliance of the speech and the detection of sensitive words include prohibited word detection, semantic violation detection and extreme word detection. The specific implementation methods are Aho-Corasick multi-pattern matching, RAG vector similarity retrieval and extreme word detection. The evaluation dimensions for expressiveness and emotion assessment include speech rate, volume, emotional persuasiveness, and fluency.
8. The e-commerce live-streaming teaching system with multimodal recognition according to claim 1, characterized in that, The system also includes a teacher-side evaluation management module, which includes automatic objective data statistics, human-computer collaborative scoring, and batch class management. The statistical items automatically compiled by the objective data include training duration, number of violations, composition pass rate, benchmark matching degree, interaction success rate, and fluency of speech. The scoring mechanism for the human-machine collaborative scoring is specifically based on the final score, which is obtained by adding the base score and the bonus score. The batch management of classes includes creating classes, setting topics, configuring weights, and exporting reports.
9. The e-commerce live-streaming teaching system with multimodal recognition according to claim 1, characterized in that, The system also includes a student-side training assistance module, which includes real-time screen guidance, interactive prompts, and automatic generation of training reports. The real-time video guidance takes three forms: mild prompts, moderate prompts, and strong intervention. The linked prompt functions include script import, font scaling, automatic scrolling, and position adjustment; The automatically generated training report includes basic information, a five-dimensional radar chart, deduction details, and improvement suggestions.
10. The e-commerce live-streaming teaching system with multimodal recognition according to claim 1, characterized in that, The system also includes an accessibility adaptation module, which consists of a visual impairment mode and a hearing impairment mode. The visually impaired mode uses screen reader software to automatically recognize and broadcast all interface elements. The hearing impairment mode uses different colored lights to flash and provide warnings for different content.
