A live interactive system, method

Abstract

This invention discloses a live streaming interactive system and method, relating to the field of live streaming interactive technology. It includes a user-end interaction module, a broadcaster-end management module, a real-time data acquisition module, an intelligent scenario-based interactive engine, a dynamic content recommendation module, a virtual asset scenario-based generation module, a real-time communication relay module, a cloud storage module, and a backend management module. This invention, by building a 3D virtual live streaming room entrance, supports users in creating or selecting virtual avatars to enter virtual scenes. It also provides multimodal interactive input methods such as voice, gestures, and facial expressions, allowing users to move beyond traditional text interaction and participate in live streaming in a more intuitive and immersive way. This enhances the connection between users, broadcasters, and the scene, improving the overall participation experience.

Description

Technical Field

[0001] This invention belongs to the field of live streaming interactive technology, and in particular relates to a live streaming interactive system and method. Background Technology

[0002] Live streaming refers to a mode of communication that transmits live video and audio content to viewers in real time via digital media platforms such as the internet, enabling "instant viewing and instant interaction." The core of live streaming lies in the real-time generation and synchronous dissemination of content, allowing viewers to watch and participate simultaneously as the event unfolds. Live streaming interaction refers to the real-time, two-way, or multi-way communication and interaction between the broadcaster and viewers, and among viewers themselves, via internet technology during the live stream. Unlike traditional one-way video playback, live streaming interaction allows viewers to participate in the live content, express opinions, ask questions, participate in activities, and even influence the progress and direction of the live stream.

[0003] With the rapid development of the live streaming industry, traditional live streaming interaction methods have many core defects: most platforms only support basic interactions such as likes, comments, and live chat, lacking immersive and scenario-based experiences, resulting in low user engagement; content recommendations rely on historical data and do not take into account users' real-time behavior and emotions, leading to poor recommendation accuracy; a single communication protocol cannot simultaneously ensure the transmission efficiency of video streams and interactive data, resulting in delayed interactive feedback; virtual gifts are disconnected from the live streaming scenario, making it difficult to stimulate users' willingness to consume; and broadcasters need to manually initiate interactions and manage the live streaming room, which is labor-intensive and affects the quality of the live streaming. Therefore, this invention proposes a live streaming interaction system and method. Summary of the Invention

[0004] This invention provides a live streaming interactive system and method. By constructing a 3D virtual live streaming room entrance, it supports users in creating or selecting virtual avatars to enter virtual scenes. It also provides multimodal interactive input methods such as voice, gestures, and facial expressions, allowing users to participate in live streaming in a more intuitive and immersive way, moving beyond traditional text interaction. This enhances the connection between users, the streamer, and the scene, improving the overall participation experience. The system uses an Attention-LSTM model to capture the temporal dependencies of user multimodal interaction data and assigns higher weights to key behavioral features. Combined with a DQN reinforcement learning model, it generates optimal interaction strategies based on the scene and user profile, ensuring that recommended content and interaction methods better match users' real-time interests and needs, reducing invalid recommendations, and encouraging users to stay on the platform and participate more actively. The system uses the WebSocket protocol to transmit text-based interaction strategy data and the QUIC protocol to transmit video streams and virtual asset data. A consistent hash load balancing mechanism is used to allocate server resources, reducing latency and packet loss during data transmission and ensuring the smooth transmission of interactive commands and virtual assets. The system ensures that product and video content can reach users quickly and stably, improving the smoothness of the live streaming process. By using a GAN model to combine live streaming scene parameters and streamer characteristics to generate exclusive virtual assets, the dynamic content recommendation module matches scenario-based products or content, making virtual gifts more contextual and personalized. E-commerce recommendations better meet users' current needs, thereby stimulating users' willingness to consume and promoting the purchase of virtual gifts and the conversion of e-commerce products. An intelligent scenario-based interaction engine automatically generates interaction strategies adapted to specific scenarios. The streamer-side management module simplifies scene settings and interaction on / off operations, reducing the workload of streamers manually planning interactions. This allows streamers to focus more on content creation and direct interaction with users, improving the quality of live streaming content and user ratings. Through the cloud storage module's three-copy redundant storage and cross-regional backup mechanism, user interaction data and virtual asset data are ensured not to be lost due to single-point failures. Simultaneously, adhering to GDPR data protection standards, user personal information is strictly encrypted and access managed, guaranteeing data reliability and user privacy security. In summary, the system solves the problems mentioned in the background technology.

[0005] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0006] The present invention provides a live streaming interactive system and method, comprising:

[0007] The module includes a user-side interaction module, a broadcaster-side management module, a real-time data acquisition module, an intelligent scenario-based interaction engine, a dynamic content recommendation module, a virtual asset scenario-based generation module, a real-time communication relay module, a cloud storage module, and a back-end management module.

[0008] The user-end interaction module is used to provide a virtual scene entrance and a multimodal interactive interface, receive and display dynamic content and virtual assets from the real-time communication relay module, and transmit user interaction operation data to the real-time data acquisition module.

[0009] The broadcaster management module is used to receive permission configuration instructions from the backend management module, manage the virtual scene and interaction strategy switch of the live broadcast room, transmit the broadcaster operation data to the real-time data acquisition module, and receive interaction strategy instructions from the intelligent scene-based interaction engine and user interaction data from the real-time communication relay module.

[0010] The real-time data acquisition module is used to collect user interaction operation data transmitted by the user terminal interaction module and anchor operation data transmitted by the anchor terminal management module, store the collected data in the cloud storage module, and transmit the real-time data to the intelligent scenario-based interaction engine.

[0011] The intelligent scenario-based interaction engine is used to receive real-time data transmitted by the real-time data acquisition module, generate interaction strategy instructions based on preset algorithms, and transmit the instructions to the dynamic content recommendation module, the virtual asset scenario generation module and the real-time communication relay module respectively. At the same time, it reads historical data from the cloud storage module for model optimization.

[0012] The dynamic content recommendation module is used to receive the interaction strategy instructions of the intelligent scenario-based interaction engine, read the corresponding dynamic content resources from the cloud storage module, generate a personalized recommendation list, and transmit the recommendation list to the real-time communication relay module.

[0013] The virtual asset scenario generation module is used to receive the interaction strategy instructions of the intelligent scenario-based interaction engine, call the preset virtual asset template and generate scenario-specific virtual assets, and transmit the generated virtual assets to the real-time communication relay module.

[0014] The real-time communication relay module is used to relay data between the user terminal interaction module, the broadcaster terminal management module, the intelligent scenario-based interaction engine, the dynamic content recommendation module, and the virtual asset scenario-based generation module, and adopts a hybrid protocol to achieve low-latency transmission of different types of data.

[0015] The cloud storage module is used to store real-time data transmitted by the real-time data acquisition module, historical data required by the intelligent scenario-based interactive engine, dynamic content resources of the dynamic content recommendation module, and virtual asset templates of the virtual asset scenario-based generation module, while providing a data query interface to the backend management module.

[0016] The backend management module is used to review the qualifications of the anchor and configure the permissions of the live broadcast room, adjust the algorithm parameters of the intelligent scene-based interaction engine, read data from the cloud storage module to generate statistical reports, and transmit the permission configuration instructions to the anchor management module.

[0017] Furthermore, the user-end interaction module includes a virtual scene entry submodule and a multimodal interaction submodule. The virtual scene entry submodule provides an access point to the 3D virtual live streaming room, through which users can create or select a virtual avatar and enter the virtual live streaming room. The multimodal interaction submodule supports multimodal input of voice, gestures, and facial expressions, and users can trigger specific interactive operations through voice commands or gestures.

[0018] Furthermore, the intelligent scenario-based interaction engine includes a user behavior profiling submodule and a scenario matching submodule. The user behavior profiling submodule constructs a real-time dynamic profile of the user based on real-time collected multi-dimensional data, and the scenario matching submodule generates an interaction strategy adapted to the scenario based on the real-time dynamic profile of the user and the current live streaming scenario tags.

[0019] Furthermore, the virtual asset scenario generation module includes an asset template library and a dynamic generation submodule. The asset template library stores various types of virtual asset templates, and the dynamic generation submodule calls the corresponding template from the asset template library and generates scenario-based virtual assets by combining the interaction strategy output by the intelligent scenario-based interaction engine with the parameters of the current live broadcast scenario.

[0020] Furthermore, the real-time communication relay module adopts a hybrid communication method using WebSocket and QUIC protocols. The WebSocket protocol is used for the transmission of text-based interactive data, while the QUIC protocol is used for the transmission of video streams and virtual scene data.

[0021] A live streaming interaction method includes the following steps:

[0022] S1. User initiates interaction request: The user initiates an interaction request in the virtual scene by making a heart shape with a gesture or sending a voice comment. The request carries the interaction type and real-time scene tag.

[0023] S2. Real-time data acquisition and storage: The real-time data acquisition module synchronously collects the user's key gestures, voice emotion features, and the anchor's scene switching operation data, and temporarily stores the raw data in the hot data storage unit.

[0024] S3. Intelligent Interaction Strategy Generation: The intelligent scenario-based interaction engine uses Attention-LSTM to extract user interests and emotional states, and then uses the DQN model to generate an interaction strategy that is adapted to the current scenario.

[0025] S4. Generating Recommendations and Gifts: The dynamic content recommendation module retrieves short videos or task lists that match the scene, and the virtual asset scene generation module generates exclusive gifts with the anchor's avatar by adjusting the template parameters through GAN.

[0026] S5. Low-latency push content: The real-time communication relay module uses WebSocket to push text and QUIC to transmit virtual assets, and uses a consistent hashing algorithm to ensure low-latency content reception across multiple terminals.

[0027] S6. Render and Feedback Data: The virtual scene on the user end renders a unique gift animation, the host end displays statistics on the execution of the interaction strategy, and both parties send feedback data such as participation time and task completion rate back to the collection module.

[0028] Furthermore, the Attention-LSTM model in S3 includes an input layer, an LSTM hidden layer, an attention layer, and an output layer; the input layer receives multimodal data collected in real time; the LSTM hidden layer captures the dependencies of time-series data and outputs a hidden state sequence; the attention layer assigns higher weights to key behavioral data in the hidden state sequence; and the output layer generates user interest preference vectors and emotional state values ​​for subsequent interaction strategy generation.

[0029] Furthermore, the DQN model in S3 adopts a reinforcement learning framework, including a state space, an action space, and a reward function. The state space includes real-time dynamic user profiles, current live streaming scene tags, and interaction popularity data. The action space includes operations such as sending virtual gifts, initiating interactive tasks, and recommending short videos. The reward function is composed of a weighted average of retention rate improvement, interaction frequency improvement, and virtual asset consumption growth. The DQN model stores historical interaction data through an experience replay mechanism, processes training fluctuations through target network separation, and generates the optimal interaction strategy adapted to the scenario.

[0030] Furthermore, the GAN model described in S4 includes a generator and a discriminator; the generator receives interaction strategy parameters and virtual asset template parameters output by the intelligent scene-based interaction engine, and generates scene-specific virtual assets by combining the anchor's avatar feature points; the discriminator compares the generated virtual assets with the real template, and feeds back the error to optimize the generator; the virtual assets generated by the GAN model are in GLB format, support real-time rendering, and the maximum generation time for a single asset is 50ms.

[0031] Furthermore, the hybrid communication method and load balancing mechanism in S5 are as follows: Text-based interactive strategy data is transmitted using the WebSocket protocol, with a maximum frame size of 1KB and a maximum latency of 50ms; video streams and virtual asset data are transmitted using the QUIC protocol, with a maximum transmission unit of 1500 bytes, and a maximum latency of 80ms when the maximum packet loss rate is 1%; the consistent hashing algorithm maps user requests to server nodes, and the virtual node mechanism handles request redirection when nodes change.

[0032] The present invention has the following advantages over the prior art:

[0033] (1) Enhanced interactive experience and optimized content recommendation: This technical solution builds a 3D virtual live streaming room entrance, which allows users to create or select virtual avatars to enter virtual scenes. It also provides multimodal interactive input methods such as voice, gestures, and facial expressions, so that users are no longer limited to traditional text interaction and can participate in live streaming in a more intuitive and immersive way. This enhances the connection between users, hosts, and scenes and improves the overall participation experience. The Attention-LSTM model captures the temporal dependencies of users' multimodal interactive data and assigns higher weights to key behavioral features. Combined with the DQN reinforcement learning model, the optimal interaction strategy is generated based on the scene and user profile. This makes the recommended content and interaction methods more in line with the user's real-time interests and needs, reduces invalid recommendations, and makes users more willing to stay on the platform and participate in the interaction.

[0034] (2) Low-latency communication transmission guarantee and enhanced commercial value conversion capability: This technical solution transmits text-based interactive strategy data through the WebSocket protocol and video streams and virtual asset data through the QUIC protocol. It is equipped with a consistent hash load balancing mechanism to allocate server resources, reduce latency and packet loss during data transmission, and ensure that interactive instructions, virtual assets and video content can reach the user terminal quickly and stably, thereby improving the smoothness of the live broadcast. By using the GAN model to generate exclusive virtual assets in combination with live broadcast scene parameters and anchor characteristics, the dynamic content recommendation module matches scenario-based products or content, making virtual gifts more scenario-based and personalized, and e-commerce recommendations more in line with the user's current needs, thereby stimulating the user's willingness to consume and promoting the purchase of virtual gifts and the conversion of e-commerce products.

[0035] (3) Improved efficiency and quality of live streaming operations, and guaranteed data security and compliance: This technical solution automatically generates interactive strategies that are adapted to the scene through an intelligent scenario-based interactive engine. The management module of the broadcaster end simplifies the scene setting and interactive switch operation, reduces the workload of broadcasters in manually planning interactions, and allows broadcasters to devote more energy to content creation and direct interaction with users, thereby improving the quality of live streaming content and users' ratings of the live stream. Through the 3-copy redundant storage and cross-regional backup mechanism of the cloud storage module, it is ensured that user interaction data, virtual asset data, etc. will not be lost due to single point of failure. At the same time, it complies with the GDPR data protection standard, strictly encrypts and manages user personal information, and ensures the reliability of data and the security of user privacy.

[0036] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of the system flow of a live interactive system and method according to the present invention;

[0039] Figure 2 This is a schematic diagram of the method flow of a live interactive system and method according to the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Specific Implementation

[0041] Please see Figure 1 As shown, the present invention provides a live streaming interactive system and method, comprising:

[0042] The module includes a user-side interaction module, a broadcaster-side management module, a real-time data acquisition module, an intelligent scenario-based interaction engine, a dynamic content recommendation module, a virtual asset scenario-based generation module, a real-time communication relay module, a cloud storage module, and a back-end management module.

[0043] Among them, the user-end interaction module is used to provide virtual scene entry and multimodal interactive interface, receive and display dynamic content and virtual assets from the real-time communication relay module, and transmit user interaction operation data to the real-time data acquisition module.

[0044] The broadcaster management module is used to receive permission configuration instructions from the backend management module, manage the virtual scene and interaction strategy switch of the live broadcast room, transmit the broadcaster operation data to the real-time data acquisition module, and receive interaction strategy instructions from the intelligent scene-based interaction engine and user interaction data from the real-time communication relay module.

[0045] The real-time data acquisition module is used to collect user interaction operation data transmitted by the user terminal interaction module and broadcaster operation data transmitted by the broadcaster terminal management module. The collected data is stored in the cloud storage module and transmitted in real time to the intelligent scenario-based interaction engine.

[0046] The intelligent scenario-based interaction engine receives real-time data transmitted by the real-time data acquisition module, generates interactive strategy instructions based on preset algorithms, and transmits the instructions to the dynamic content recommendation module, the virtual asset scenario generation module, and the real-time communication relay module respectively. At the same time, it reads historical data from the cloud storage module for model optimization.

[0047] The dynamic content recommendation module receives interaction strategy instructions from the intelligent scenario-based interaction engine, reads the corresponding dynamic content resources from the cloud storage module, generates a personalized recommendation list, and transmits the recommendation list to the real-time communication relay module.

[0048] The virtual asset scenario generation module is used to receive the interaction strategy instructions of the intelligent scenario-based interaction engine, call the preset virtual asset template and generate scenario-specific virtual assets, and transmit the generated virtual assets to the real-time communication relay module.

[0049] The real-time communication relay module is used to relay data between the user-end interaction module, the broadcaster-end management module, the intelligent scenario-based interaction engine, the dynamic content recommendation module, and the virtual asset scenario-based generation module. It adopts a hybrid protocol to achieve low-latency transmission of different types of data.

[0050] The cloud storage module is used to store real-time data transmitted by the real-time data acquisition module, historical data required by the intelligent scenario-based interactive engine, dynamic content resources of the dynamic content recommendation module, and virtual asset templates of the virtual asset scenario-based generation module, while providing a data query interface to the backend management module.

[0051] Among them, data is stored in a hierarchical manner:

[0052] Hot data (real-time interactive data, virtual assets): stored on SSD cloud disk, access latency ≤10ms;

[0053] Cold data (historical live stream records, user profiles): stored in object storage, reducing costs by 50%;

[0054] Data backup: Employs a 3-replica redundancy strategy, ensuring data reliability ≥99.9999%; supports cross-regional backup (e.g., East China → North China), with disaster recovery time ≤1 hour.

[0055] The backend management module is used to review the qualifications of the anchors and configure the permissions of the live broadcast room, adjust the algorithm parameters of the intelligent scene-based interaction engine, read data from the cloud storage module to generate statistical reports, and transmit the permission configuration instructions to the anchor management module.

[0056] Among them, the anchor management supports anchor qualification verification (ID card OCR recognition, face recognition verification) and live room permission configuration (interaction strategy switch, virtual scene selection).

[0057] Data statistics: Generate multi-dimensional reports (user retention rate, interaction frequency, virtual asset consumption), and support visualization (line chart, bar chart);

[0058] Algorithm configuration: Allows administrators to adjust the weights of the DQN reward function and the training parameters of Attention-LSTM, and supports model version management (retaining the 5 most recent versions).

[0059] The user-side interaction module includes a virtual scene entry submodule and a multimodal interaction submodule. The virtual scene entry submodule provides access to the 3D virtual live streaming room. Users can create or select a virtual avatar and enter the virtual live streaming room through this entry. The multimodal interaction submodule supports multimodal input of voice, gestures, and facial expressions. Users can trigger specific interactive operations through voice commands or gestures.

[0060] The virtual scene entry submodule uses WebGL 2.0 to render 3D virtual scenes in real time, supporting 1080P@60fps output. Virtual avatar customization dimensions include hairstyles (20+ styles), clothing (50+ sets), facial expressions (15+ animated expressions), and accessories (30+ categories). Users can upload photos to trigger AI to generate 3D models (based on StyleGAN2 facial feature extraction), adjust parameters, and save to cloud storage (GLB format, single model size ≤ 5MB). It's worth noting that traditional virtual avatars are mostly preset templates, with low user customization and complex generation processes. This module uses StyleGAN2 to quickly generate 3D avatars from photos, enhancing user immersion with personalized avatars. Specifically:

[0061] Face preprocessing: After the user uploads a photo, the system uses the MTCNN algorithm (MTCNN is a multi-task face detection and alignment model that achieves efficient and accurate face processing through a three-stage cascaded network: P-Net: a lightweight convolutional network that quickly scans the image to generate face candidate boxes and make preliminary shape predictions; R-Net: performs secondary filtering on the candidate boxes, corrects the bounding box positions, and eliminates non-face regions; O-Net: accurately outputs the face bounding box, 5 key feature points (eyes, nose, mouth) and confidence scores, and completes face alignment, such as correcting tilt angles and unifying sizes) to complete face detection and alignment, extract key feature points such as eyes, nose, and mouth, and unify image size and angle, laying the foundation for subsequent feature extraction;

[0062] StyleGAN2 Feature Mapping: The preprocessed photo is input into the StyleGAN2 model, which uses a multi-layer convolutional network (a feature extraction structure composed of multiple stacked convolutional layers: bottom convolutional layers extract low-level image features (edges, color, texture) using small kernels (e.g., 3×3); middle convolutional layers combine low-level features to extract local structural features (e.g., eye shape, eyebrow outline); high-level convolutional layers fuse middle-level features to extract abstract global features (e.g., face shape, facial proportions, overall style). StyleGAN2 also introduces residual connections and adaptive upsampling to enhance feature transfer efficiency and avoid gradient vanishing) and style modulation (StyleModu). StyleGAN's core innovative technology, based on Adaptive Instance Normalization (AdaIN), is used to normalize the feature maps of each convolutional layer during generation, replacing their mean and variance with the mean and variance of the target style. This preserves the content information of the input image (such as the user's facial contours) while injecting style information (such as the texture of a "warrior's" armor or an ancient-style hairstyle in a game scene). StyleGAN2 optimizes the hierarchical control of style modulation, supporting independent adjustment of style parameters for different parts (such as face, hairstyle, and clothing). It also extracts personalized facial feature vectors (including face shape, facial proportions, skin tone, and hairstyle texture) to capture the user's unique facial identifiers.

[0063] 3D Model Generation and Lightweighting: The feature vectors output by StyleGAN2 are mapped to the base model of the preset 3D template library, and the facial parameters of the template (such as cheekbone height and eye size) are dynamically adjusted to match the user's features. Then, through polygon simplification and texture compression techniques, the model size is controlled to within 5MB and converted to GLB format (a general 3D format that supports real-time rendering and is compatible with mainstream virtual scene engines).

[0064] Dynamic scene adaptation: Based on the live streaming scene tags (such as "game team battle" or "streamer's birthday"), the clothing, accessories and posture parameters of the virtual avatar are automatically adjusted (for example, adding armor and weapons in the game scene and adjusting the standing posture to a combat posture), so as to achieve a natural integration of the avatar and the scene, which greatly reduces the time for users to enter the virtual scene (≤2 seconds) and enhances the immersive experience.

[0065] Multimodal interaction submodule

[0066] Gesture recognition: Based on the Google MediaPipe hand keypoint detection algorithm, it adopts a two-stage model of "palm detection + keypoint tracking".

[0067] Hand detection: Quickly locate the hand region in an image using a lightweight CNN model and output bounding boxes;

[0068] Key point tracking: For the detected palm area, a regression model is used to capture 21 3D hand joints in real time (covering key positions such as fingertips, knuckles, and wrists), with joint coordinate accuracy reaching sub-pixel level;

[0069] Gesture classification: Based on the relative position and angle changes of 21 key points, a pre-trained classifier is used to identify 10+ preset gestures (heart gesture, thumbs up, sword gesture, etc.). The classification logic is combined with geometric features (e.g., the angle between the thumb and index finger when making a heart gesture is ≤30°).

[0070] Specifically, this manifests as: ① The camera captures video frames of the user's hand in real time;

[0071] ②Preprocessing: Convert the frames to grayscale and normalize their size (to adapt to the model input);

[0072] ③MediaPipe model inference: Outputs the 3D coordinates of 21 hand joints;

[0073] ④ Gesture matching: Compare the key feature with the preset gesture template and output the recognition result;

[0074] ⑤ Interactive trigger: If recognition is successful, send instructions to the live streaming scene (such as triggering special effects or tasks) within ≤80ms.

[0075] This greatly improves the recognition accuracy (≥95%), avoids false triggers (e.g., a user's unintentional hand gesture will not be recognized as a like); has low response latency (≤80ms), strong synchronization with live content (e.g., when a user is sparring in a game live stream, the streamer's character immediately launches an attack); and has high contextual interaction (a heart gesture triggers a full-screen heart effect, a like unlocks the streamer's exclusive voice feedback, and a sword sparring starts a team PK task), significantly enhancing user engagement.

[0076] Speech emotion recognition: Employs a CNN-LSTM hybrid model, combining the time and frequency domain features of the speech signal.

[0077] Mel-frequency conversion: The speech signal is divided into frames (25ms frame length, 10ms frame shift) and converted into a 256-dimensional Mel-frequency spectral feature vector through a Mel filter bank. This simulates the nonlinear perception of frequency by the human ear. It is worth noting that the Mel filter bank is a key tool for simulating the frequency perception characteristics of the human auditory system, and its core is based on the nonlinear frequency mapping of the Mel scale.

[0078] The essence of the Mel scale: The human ear has a higher frequency resolution for low-frequency sounds (e.g., a 100Hz difference is easily distinguishable), but a lower resolution for high-frequency sounds (e.g., a 1000Hz difference is difficult to distinguish). The Mel scale converts linear frequencies into a non-linear scale consistent with auditory perception.

[0079] The formula is: ,in It is a linear frequency (Hz). After conversion, the low-frequency band has denser markings, while the high-frequency band has sparser markings.

[0080] The filter bank is designed as follows: it consists of a series of triangular filters (256 in total, corresponding to the target feature dimension). The center frequency of each filter is evenly distributed according to the Mel scale. The filter shape is an isosceles triangle, and the vertex of the adjacent filter overlaps with the end point of the base of the previous filter (overlap rate of about 50%-70%) to ensure gapless spectrum coverage. The bandwidth of the filter increases with the center frequency to match the low resolution characteristics of the human ear for high frequencies.

[0081] The process of generating 256-Vimel spectral feature vectors (taking speech emotion recognition as an example):

[0082] 1. Speech signal preprocessing

[0083] Framing: The continuous speech signal is divided into short frames (e.g., 25ms / frame, each frame contains 400 samples at a sampling rate of 16kHz), and the frame shift is set to 10ms (overlap rate of 60%) to preserve timing information.

[0084] Windowing: Applying a Hamming window to each frame of signal reduces spectral leakage at frame boundaries (Hamming window formula: ,in (frame length).

[0085] 2. Frequency Domain Transformation (FFT)

[0086] Perform a Fast Fourier Transform (FFT) (such as a 512-point FFT) on the windowed signal of each frame to convert the time-domain signal into a frequency-domain power spectrum, and obtain the energy distribution at the linear frequency.

[0087] 3. Mel Filter

[0088] The power spectrum is input into 256 Mel-triangle filters, with each filter retaining only the energy of the corresponding Mel frequency band. The calculation method is as follows: ,in It is the output energy of the m-th filter. It is the spectrum value after FFT. It is the weight of the m-th filter at frequency point k;

[0089] 4. Logarithmic compression

[0090] Take the natural logarithm of the output energy of each filter ( It simulates the logarithmic perception of sound intensity by the human ear, while compressing the dynamic range and reducing noise interference.

[0091] 5. Feature Vector Generation

[0092] Each frame of speech corresponds to a 256-dimensional vector. Each dimension of the vector is the logarithmic energy value of a Mel filter, which ultimately forms a two-dimensional feature matrix of "time × 256" (e.g., 1 second of speech contains 100 frames, and the matrix size is 100 × 256).

[0093] The role of voice emotion recognition

[0094] Based on the CNN-LSTM hybrid model mentioned by the user, the core value of Mel spectral features is reflected in three aspects:

[0095] 1. Accurately capture emotion-related features

[0096] Key cues of vocal emotion (such as a surge in high-frequency energy during excitement, rapid and focused speech during anger, and a flat spectrum during calmness) are more prominent in the Mel spectrum:

[0097] Excitement: High-frequency band (e.g., 2-4kHz) filter energy increases significantly;

[0098] Anger: Energy is generally enhanced across all frequency bands, and there are dramatic energy changes between frames;

[0099] Calm emotions: High energy content in the low frequency range (such as 0.5-1kHz) and small spectral fluctuations.

[0100] 2. Adapting the feature extraction logic to CNN-LSTM

[0101] The input to the CNN layer: the “frequency dimension (256 dimensions)” of the Mel spectrum corresponds to the spatial features of the CNN, and local spectral patterns (such as frequency combinations specific to emotions) are extracted through convolutional kernels.

[0102] The input to the LSTM layer: The “time dimension (frame sequence)” of the Mel spectrum corresponds to the temporal features of the LSTM, capturing the trend of emotion changes over time (such as the gradual change in energy from calm to excitement).

[0103] 3. Improve recognition accuracy

[0104] Compared to linear spectrum, Mel spectrum filters out redundant high-frequency information and focuses on auditory-sensitive emotional features. This process transforms speech signals into highly recognizable emotional features by simulating auditory perception, providing a reliable input basis for subsequent model inference.

[0105] CNN Feature Extraction: Local texture features (such as spectral peaks and frequency distribution) of the Mel spectrum are extracted using 3 convolutional layers.

[0106] LSTM temporal modeling: Using a 2-layer bidirectional LSTM to capture the temporal dependence of speech signals (such as the rhythm of emotional changes and intonation fluctuations);

[0107] Emotion Classification: Four types of emotion results (excitement, calmness, anger, sadness) are output through a fully connected layer.

[0108] Specifically, this manifests as: ① The microphone collects the user's voice signal;

[0109] ②Preprocessing: noise reduction (using spectral subtraction) and normalization (eliminating volume differences);

[0110] ③ Mel spectrum generation: Convert to a 256-dimensional feature vector;

[0111] ④ Model inference: CNN-LSTM outputs the sentiment classification probability;

[0112] ⑤ Strategy Trigger: Push appropriate live interactive content based on the highest probability emotion.

[0113] This improves the accuracy of emotion perception (≥90%), effectively distinguishing user emotions (e.g., increased tone and large frequency fluctuations when a user is excited); it pushes popular tasks (e.g., "Everyone Sends Gifts") when the user is excited, recommends soothing short videos (e.g., clips of a host singing) when the user is calm, and triggers comforting prompts from the host when the user is angry; through emotion-driven content recommendations, it increases the average user dwell time.

[0114] Facial Expression Interaction: A 68-Facial Feature Point Detection Algorithm Based on Dlib

[0115] Using HOG (Histogram of Oriented Gradients) feature extraction and regression tree model, 68 key facial feature points (covering areas such as eyes, eyebrows, nose, mouth, and chin) were located. Specifically, Dlib first calculated the gradient of the face image to obtain the direction and intensity of each pixel; then, the image was divided into small cells, and the gradient orientation histogram of each cell was calculated; finally, the cells were normalized into blocks to generate HOG feature vectors—this vector can effectively capture key structural information such as the edges and contours of the face, providing a foundation for subsequent feature point localization.

[0116] A multi-round iterative cascaded regression tree is used to optimize the feature point location: initially, it is assumed that the feature point is located at a preset position of "average face"; in each round of the regression tree, the position offset is predicted based on the HOG features around the current feature point; the feature point coordinates are gradually corrected until they converge to the accurate position (error is less than the threshold).

[0117] 68 points cover the core area of ​​the face, and their specific distribution and expression associations are as follows:

[0118]

[0119] By calculating the relative displacement / distance changes of these feature points (such as the degree of upward movement of the corners of the mouth and the degree of eyelid opening and closing), the type of expression (smiling, frowning, squinting, etc.) can be determined.

[0120] Expression detection: Compare the displacement of feature points with the baseline state to trigger a preset expression threshold.

[0121] Smile: The y-coordinate of the corner of the mouth (49, 55) increases by ≥5 pixels;

[0122] Squinting: Reduce the vertical distance between the eyelid points (37-41) by ≥3 pixels;

[0123] Frowning: The y-coordinate of the eyebrow tail point (21, 24) is shifted down by ≥2 pixels;

[0124] Virtual avatar update: Map facial expression results to virtual avatar parameters, such as:

[0125] Smile → Adjust the upward curve of your mouth + trigger the blinking animation of your eyes;

[0126] Frowning → Adjust the "downward movement" of the eyebrows;

[0127] The entire process is controlled within ≤100ms to ensure real-time synchronization.

[0128] Specifically, this manifests as: ① The camera captures video frames of the user's face;

[0129] ②Dlib model inference: Outputs 2D coordinates of 68 facial feature points;

[0130] ③ Expression judgment: Calculate the relative displacement of feature points (e.g., if the difference in y-coordinate of the corners of the mouth is ≥5 pixels, it is judged as a smile);

[0131] ④ Virtual Avatar Update: Synchronously adjust the virtual avatar's facial expression parameters within ≤100ms;

[0132] ⑤ Scene Feedback: The virtual avatar's expressions are displayed in real time during the live stream.

[0133] This allows users to synchronize their facial expressions with their virtual avatars, enhancing their sense of immersion (e.g., when a user frowns, the virtual avatar displays worry, increasing emotional resonance). No manual operation is required, and the interactions triggered by facial expressions are more in line with the user's real emotions (e.g., when a user laughs, the virtual avatar throws out a surprise gift). The Dlib model is lightweight, with a single frame processing time of ≤20ms, which does not affect the smoothness of the live stream.

[0134] The three technologies work together to build a full-dimensional interactive link of "gestures + voice + facial expressions", which upgrades live interaction from "passive response" to "active perception", significantly improving the user's immersive experience and participation.

[0135] The intelligent scenario-based interaction engine includes a user behavior profiling submodule and a scenario matching submodule. The user behavior profiling submodule constructs a real-time dynamic profile of the user based on real-time collected multi-dimensional data, while the scenario matching submodule generates an interaction strategy that is adapted to the scenario based on the real-time dynamic profile of the user and the current live streaming scenario tags.

[0136] The intelligent scenario-based interaction engine serves as the decision-making hub connecting users' multimodal behaviors with dynamic scenarios in live streaming. Its core objective is to generate precisely tailored interaction strategies based on real-time user behavior and scenario tags. Its core algorithm is a multimodal fusion real-time scenario interaction decision-making algorithm. This algorithm integrates the Attention-LSTM real-time profile model and the DQN reinforcement learning strategy model. Through a closed-loop process of "data input → feature fusion → profile construction → strategy generation → feedback optimization," it achieves real-time, personalized, and scenario-adaptive interaction strategies.

[0137] Among them, the Attention-LSTM real-time dynamic portrait model

[0138] Attention mechanism: For the multimodal feature sequence within a 10-second time window, calculate the attention weight for each time step (e.g., the weight of gesture interaction during team battles is 0.8, and the weight of ordinary comments is 0.2).

[0139] LSTM hidden layers: capture temporal features (such as the user's emotional change from calm to excitement);

[0140] Output layer: Generates a 128-dimensional interest preference vector and a 16-dimensional emotion state vector, providing the core basis for strategy generation.

[0141] DQN reinforcement learning strategy model

[0142] State representation: The user profile (144-dimensional), scene label (one-hot encoded, such as "team battle" as [1,0,0]), and interaction popularity (4-dimensional: number of online users / number of gifts / number of comments / number of likes) are concatenated into a 150-dimensional state vector;

[0143] Action selection: An ε-greedy strategy (ε=0.1) is adopted to balance exploration and exploitation, with a 10% probability of randomly selecting a new strategy and a 90% probability of selecting the current optimal strategy;

[0144] Value function update: Update the Q value using the Bellman equation, the formula is as follows:

[0145] ;

[0146] The technical principle of the intelligent scenario-based interactive engine is based on two core logics: multimodal data fusion and real-time decision-making closed loop.

[0147] Multimodal data heterogeneous fusion: Transform heterogeneous data such as gestures, voice, text, and behavior into feature vectors of a unified dimension, and capture key behaviors in time series through Attention-LSTM to build dynamic user profiles;

[0148] Reinforcement learning real-time decision making: Using user profiles and scene tags as input, the DQN model explores the optimal solution in the interaction strategy space to maximize user engagement and business value;

[0149] Feedback-loop iteration: Continuously update model parameters based on user interaction data to achieve dynamic optimization of the strategy.

[0150] By leveraging multimodal fusion and reinforcement learning technologies, the live streaming interaction has been upgraded from "passive response" to "active adaptation," providing live streaming platforms with efficient tools for user growth and monetization.

[0151] The implementation process of the intelligent scenario-based interactive engine consists of four core steps: data preprocessing, real-time profile construction, strategy generation, and feedback optimization. Details of each step are as follows:

[0152] 1. Data Preprocessing

[0153] The four types of input data are standardized and transformed into feature vectors that the model can recognize:

[0154] Gesture images: normalized to 224×224 pixels, and 512-dimensional feature vectors were extracted using ResNet50;

[0155] Speech signal: 16kHz sampling rate, converted to 128×128 megapixel spectrum, and 256-dimensional feature vector extracted by CNN;

[0156] Text data: The Word2Vec model (training corpus of 1 million+ live stream comments) was used to convert the text into 300-dimensional word vectors.

[0157] Behavioral data: normalized to the [0,1] range (e.g., number of likes / total number of viewers, comment frequency / live broadcast duration).

[0158] 2. Real-time dynamic user profile construction (user behavior profile submodule)

[0159] Building real-time user profiles based on multimodal feature sequences:

[0160] Model architecture: Attention-LSTM (input layer → LSTM hidden layer → attention layer → fully connected layer);

[0161] Input: A sequence of multimodal feature vectors within a 10-second time window;

[0162] Output: Interest preference vector (e.g., MOBA games account for 0.7%, beauty content accounts for 0.3%).

[0163] Emotional state values ​​(e.g., arousal level 0.85, calmness level 0.15);

[0164] Core improvement: An attention layer is introduced to give higher weight to high-value behaviors (such as gesture interactions in team battles), resulting in higher accuracy compared to traditional LSTM profiling.

[0165] 3. Policy Generation (Scene Matching Submodule - DQN Reinforcement Learning Model)

[0166] Using user profiles and scene tags as input, generate the optimal interaction strategy:

[0167] State space: User profile vector (interests + emotions), live streaming scene tags (such as "team battle" and "birthday"), current interaction popularity (number of online users, frequency of gift sending);

[0168] Action space: Includes three types of strategies (10 sub-actions in total):

[0169] Virtual gift delivery (5 types: regular gifts / special effects gifts / customized gifts, etc.);

[0170] Initiate interactive tasks (3 types: team battle assistance / birthday wishes / quiz lottery, etc.);

[0171] Short video recommendations (2 types: game highlights / streamer behind-the-scenes footage, etc.);

[0172] Reward function: Taking into account user retention, interaction frequency, and commercial value, the formula is as follows:

[0173]

[0174] in, (Retained) (interactive), (Consumption - can be dynamically adjusted);

[0175] Training process:

[0176] Experience replay: The ReplayBuffer size is 100,000 records to avoid the influence of sample correlation;

[0177] Target network separation: The target network is updated every 1000 steps to stabilize the training process;

[0178] Convergence criteria: After ≥5000 training epochs, the accuracy after policy convergence is ≥88%;

[0179] Core improvement: Compared to traditional static rule strategies, the interaction frequency is greatly increased through a composite reward function and feedback loop.

[0180] 4. Feedback Optimization

[0181] Constructing a closed loop for model iteration:

[0182] User interaction data (interaction duration, task completion rate, and number of gifts sent) is collected every 5 minutes.

[0183] Update the parameters of the Attention-LSTM and DQN models using gradient descent;

[0184] The model iteration cycle is ≤1 hour to ensure that the strategy is synchronized with changes in user behavior.

[0185] The virtual asset scenario generation module includes an asset template library and a dynamic generation sub-module. The asset template library stores various types of virtual asset templates, and the dynamic generation sub-module calls the corresponding template from the asset template library and generates scenario-based virtual assets based on the interaction strategy output by the intelligent scenario-based interaction engine and the parameters of the current live broadcast scenario.

[0186] 1. Asset Template Library: Basic Support for Parametric Design

[0187] The template library is a dynamically generated "material pool" that uses parametric design and structured storage.

[0188] Categorization and Parametric Design: Covering three major categories: gifts (100+), props (80+), and badges (50+), each template is a parametric structure (e.g., the "birthday cake" template contains 15 adjustable parameters such as radius, number of layers, number of candles, color, and animation effects), supporting flexible customization;

[0189] Storage and association: Template parameters are stored in JSON format and associated with GLB format 3D model files (lightweight, industry standard format that supports real-time rendering).

[0190] Reusability: Template parameters are decoupled from 3D models, and the same model can generate assets with different styles by adjusting parameters (e.g., red cake → blue cake, only the color field of the JSON needs to be modified).

[0191] Dynamically generated sub-modules: the core of scenario-based and personalized design

[0192] Real-time asset customization is achieved based on GAN models and scene labeling:

[0193] Personalized modification of GAN model: Combining 68 feature points of the anchor's face detected by Dlib, the anchor's avatar is embedded into the template (such as the avatar decoration on the top of the "birthday cake") through the GAN model. The generator of GAN is responsible for adjusting the texture / geometric parameters of the template, and the discriminator ensures that the embedding effect is natural (consistent with the style of the template).

[0194] Scene tag-driven adaptation: Adjusting asset attributes based on live stream scene tags (such as "game team battle" or "e-commerce discount").

[0195] Game scenario: Add "buff effect" descriptions to gifts (such as "team attack power +10% after use"), modify the effect field of the JSON and update the effect textures of the GLB model;

[0196] E-commerce scenario: Add "discount labels" (such as "50% off for a limited time") to props and overlay dynamic text textures on GLB models;

[0197] Real-time rendering optimization: The generated GLB assets support real-time GPU rendering. The complexity is reduced through LOD (Level of Detail) technology, and the rendering time of a single asset is ≤50ms, which meets the low latency requirements of live streaming.

[0198] In the specific implementation process, template selection is as follows: based on the needs triggered by user interaction (such as a user sending a "birthday wishes" command), a corresponding template (such as "birthday cake") is matched from the template library.

[0199] Personalized modification: The Dlib model was used to detect 68 feature points on the anchor's face and extract the avatar region;

[0200] The GAN model adjusts template parameters in real time: embeds the avatar into a specified position in the template (such as the top of a cake), and generates personalized asset JSON parameters and a GLB model;

[0201] Scene adaptation: Read the current live stream scene tags (e.g., "game team battle");

[0202] Adjust asset attributes: Add effect:"team_attack_buff" to the JSON and update the effects components of the GLB model;

[0203] Rendering output: The generated GLB assets are pushed to the user's end and rendered in real time (≤50ms) through the WebGL / Unity engine, allowing users to interact directly (such as clicking to send a gift).

[0204] The real-time communication relay module adopts a hybrid communication method using WebSocket and QUIC protocols. WebSocket is used for transmitting text-based interactive data, while QUIC is used for transmitting video streams and virtual scene data.

[0205] The real-time communication relay module serves as the data transmission hub of a live interactive system, its core function being to solve the problem of efficient, low-latency, and stable transmission of various types of interactive data (text, video, virtual assets). Its design philosophy is based on a dual-core approach of "protocol adaptation + intelligent routing."

[0206] Protocol adaptation: Select the optimal network protocol (WebSocket / QUIC / HTTP / 3) for different data transmission characteristics.

[0207] Intelligent routing: It achieves load balancing of requests through a consistent hashing algorithm, ensuring stability in high-concurrency scenarios.

[0208] The consistent hashing algorithm works as follows:

[0209] Ring space mapping: Maps server nodes and user requests to a 2^32-bit ring space (by calculating the hash value of node IP / user ID using a hash function).

[0210] Request routing rule: The hash value of the user request is used to find the nearest server node clockwise on the ring, which is then used as the target routing node;

[0211] Virtual node optimization: Each physical node corresponds to 10-20 virtual nodes (generated by node IP + suffix), which are evenly distributed on the ring to avoid "data skew" (excessive load on a certain node).

[0212] 2. Technical Parameter Analysis

[0213] 10+ server nodes: Supports horizontal scaling to handle large-scale live streaming scenarios with tens of thousands of online users;

[0214] Maximum concurrent connections per node ≥ 10,000: meets high concurrency requirements (e.g., 10 nodes can support 100,000 concurrent connections).

[0215] The module's overall architecture is a layered design:

[0216] Data access layer: Receives various types of data from the user terminal and distributes them to the corresponding protocol processing channels according to the data type;

[0217] Protocol processing layer: Encodes and decodes data from different protocols, performs flow control, and retransmits lost packets;

[0218] Load balancing layer: routes requests to the optimal server node using consistent hashing;

[0219] Data forwarding layer: pushes the processed data to the target end (broadcaster end / user end).

[0220] The module employs a differentiated protocol strategy for three core data types to match their respective transmission requirements:

[0221] 1. Text interaction data (comments, likes): WebSocket protocol

[0222] Data characteristics: high frequency, small volume (single comment ≤ 500 characters), strong real-time requirement;

[0223] Reason for protocol selection: WebSocket is a full-duplex long-connection protocol, avoiding the frequent handshake overhead of HTTP short connections, and is suitable for frequent small data transmissions;

[0224] Technical parameters analysis:

[0225] Port 8080: A commonly used unencrypted port with strong firewall penetration capabilities, reducing the user's connection failure rate;

[0226] Frame size ≤ 1KB: Matches the small size of text data, reducing transmission latency (small frames can be processed quickly).

[0227] Latency ≤ 50ms: Ensures "second-level response" for interactive operations such as comments and likes, improving user experience.

[0228] 2. Video stream / virtual scene data: QUIC protocol

[0229] Data characteristics: high bandwidth (video stream bitrate ≥ 1Mbps), high real-time requirements (frame interval ≤ 33ms), and strong resistance to packet loss;

[0230] Reasons for protocol selection: QUIC is based on the UDP protocol, which solves the "head-of-line blocking" problem of TCP, supports 0-RTT handshake (reducing connection establishment latency), multiplexing (transmitting multiple streams of data on the same connection), and forward error correction (FEC) (anti-packet loss).

[0231] Technical parameters analysis:

[0232] Port 443: Same as the HTTPS port, bypassing ISP restrictions on UDP ports and improving connection success rate;

[0233] Maximum Transmission Unit (MTU) 1500 bytes: Matches the standard Ethernet frame size, avoiding IP fragmentation (fragmentation increases the risk of packet loss and reassembly delay).

[0234] With a packet loss rate of ≤1%, the latency is ≤80ms: QUIC's FEC mechanism can restore data directly without retransmission in low packet loss scenarios, ensuring the smoothness of video streams / virtual scenes.

[0235] 3. Virtual asset files: HTTP / 3 protocol

[0236] Data characteristics: large file size (virtual gift GLB file ≥ 1MB), some delay is acceptable but reliable download is required;

[0237] Reasons for protocol selection: HTTP / 3 is based on the QUIC protocol, inheriting its low latency and packet loss resistance characteristics, while also supporting breakpoint resumption (solving the problem of large file download interruption) and multiplexing (parallel download of multiple assets).

[0238] Technical parameters analysis:

[0239] Resume download from where you left off: If the network is interrupted when a user is downloading virtual assets, the download can resume from where it left off after the connection is restored, avoiding repeated bandwidth consumption;

[0240] Download speed ≥ 1MB / s: Ensure that virtual assets (such as 3D gifts) are loaded within 2 seconds without affecting the user interaction process.

[0241] Please see Figure 2 As shown, a live streaming interaction method includes the following steps:

[0242] S1: User initiates multimodal interaction request

[0243] Users enter the 3D live streaming room through the virtual scene entry submodule, select / generate a virtual avatar (generation time ≤ 2 seconds); then initiate an interaction request by making a heart gesture (Dlib detects 68 facial feature points to associate with the interaction intent) or sending a voice comment, with the request carrying the interaction type (such as gift, task) and real-time scene tag (such as game team battle, streamer's birthday); the user-side interaction module encapsulates the operation instructions into JSON format (size ≤ 200B) and sends them to the real-time communication relay module.

[0244] S2: Multi-dimensional real-time data acquisition and preprocessing

[0245] The real-time data acquisition module synchronously collects three types of data at a frequency of 10 times per second:

[0246] User behavior data: number of likes, comment content, gift sending records (structured data);

[0247] Device input data: Gesture key point image (224×224 pixels), 1-second audio clip (for extracting emotional features), facial expression feature points (68, unstructured data).

[0248] Scene tag data: live stream theme (e.g., "MOBA team battle"), current scene timestamp (e.g., "00:15 team battle begins", semi-structured data);

[0249] The collected data is compressed (compression rate ≥50% for structured data and ≥30% for unstructured data) and temporarily stored in the hot data storage unit.

[0250] S3: Intelligent Scenario-Based Interactive Strategy Generation

[0251] The intelligent scenario-based interaction engine performs the following operations:

[0252] Multimodal feature fusion: Gesture image features (512-dimensional), voice emotion features (256-dimensional), and text comment vectors (300-dimensional) are concatenated into a 1024-dimensional fusion vector;

[0253] User profile construction: Extract user interest preferences (e.g., MOBA games account for 0.75%) and emotional state (e.g., excitement level 0.9) through the Attention-LSTM model (input fused vector sequence, time window 10 seconds).

[0254] Strategy Generation: The DQN reinforcement learning model uses user profile vectors, real-time scene labels, and current interaction popularity (number of online users, frequency of gift sending) as the state space to generate interaction strategies that are suitable for the scene (such as team battle bonus gift pack push).

[0255] Strategy encapsulation: Convert strategy instructions into JSON format (size ≤ 500B) and send them to the dynamic content recommendation module and the virtual asset scenario generation module.

[0256] S4: Dynamic Content Recommendation and Contextualized Generation of Virtual Assets

[0257] Dynamic content recommendation module: retrieves short video resources that match the scene (such as "MOBA team battle strategy") and generates 3 personalized recommendation lists;

[0258] Virtual asset scene generation module: Call the corresponding template (such as "Benefit Pack"), adjust the template parameters through the GAN model (such as embedding the anchor's avatar), generate exclusive virtual assets (time ≤ 100ms), and output format is GLB (supports real-time rendering).

[0259] S5: Multi-protocol low-latency content delivery

[0260] The real-time communication relay module uses a hybrid protocol to push content:

[0261] Text-based interactive strategies (such as task instructions) are pushed via the WebSocket protocol (port 8080, latency ≤50ms).

[0262] Virtual assets (such as exclusive gift packs) are transmitted via the QUIC protocol (port 443, MTU=1500 bytes, latency ≤80ms).

[0263] We recommend downloading short videos via HTTP / 3 (speed ≥ 1MB / s, supports resuming interrupted downloads);

[0264] Requests are routed to the optimal server node using a consistent hashing algorithm, ensuring low-latency reception across multiple devices (maximum concurrent connections per node ≥ 10000).

[0265] S6: Feedback Data Acquisition and Model Closed-Loop Optimization

[0266] The user-side renders a unique gift animation (single asset rendering time ≤ 50ms), while the broadcaster-side displays interactive strategy execution statistics (such as task completion rate and gift sending volume).

[0267] User participation data (interaction duration, task completion rate, virtual asset consumption records) is collected in real time and pushed to the intelligent scenario-based interaction engine every 5 minutes;

[0268] The engine updates the parameters of the Attention-LSTM and DQN models using the gradient descent algorithm, with a model iteration cycle of ≤1 hour, thereby achieving dynamic optimization of the strategy.

[0269] In specific implementation processes, such as e-commerce live streaming scenarios: virtual showroom interaction

[0270] Scene background: The host is live streaming a "digital product launch event" in a virtual "future technology exhibition hall" with 800 online users. The current scene tag is "new mobile phone launch".

[0271] Interactive process:

[0272] 1. 00:10: User B initiates interaction via voice command ("Want to see phone details");

[0273] 2. 00:10.2: Real-time collection of user B's voice emotion (calmness level 0.85) and behavioral data (browsing products twice in the last 3 minutes);

[0274] 3. 00:10.5: Intelligent engine generation strategy: Push "360° virtual model of mobile phone" + initiate "reservation lottery" task;

[0275] 4. 00:10.8: The virtual asset module calls the "360° Model" template, embeds mobile phone parameters (such as "Snapdragon 8 Gen3") to generate interactive props;

[0276] 5. 00:11: User B views the phone details through a virtual model (supports rotation and zoom) and completes the reservation for the lucky draw.

[0277] 6. 00:12: Feedback data shows that user B's product clicks increased by 30% and reservation rate increased by 20%.

[0278] The Attention-LSTM model in S3 includes an input layer, an LSTM hidden layer, an attention layer, and an output layer. The input layer receives multimodal data collected in real time. The LSTM hidden layer captures the dependencies of time-series data and outputs a hidden state sequence. The attention layer assigns higher weights to key behavioral data in the hidden state sequence. The output layer generates user interest preference vectors and emotion state values ​​for subsequent interaction strategy generation.

[0279] In S3, the DQN model adopts a reinforcement learning framework, including a state space, an action space, and a reward function. The state space contains real-time dynamic user profiles, current live streaming scene tags, and interaction popularity data. The action space includes operations such as sending virtual gifts, initiating interactive tasks, and recommending short videos. The reward function is composed of a weighted average of retention rate improvement, interaction frequency improvement, and virtual asset consumption growth. The DQN model stores historical interaction data through an experience replay mechanism, processes training fluctuations through target network separation, and generates the optimal interaction strategy adapted to the scenario.

[0280] The GAN model in S4 includes a generator and a discriminator. The generator receives interaction strategy parameters and virtual asset template parameters from the intelligent scene-based interaction engine, and generates scene-specific virtual assets by combining the anchor's avatar feature points. The discriminator compares the generated virtual assets with the real template and provides feedback on the error to optimize the generator. The virtual assets generated by the GAN model are in GLB format, support real-time rendering, and the maximum generation time for a single asset is 50ms.

[0281] Specifically, the hybrid communication method and load balancing mechanism in S5 are as follows: the WebSocket protocol is used to transmit text-based interactive strategy data, with a maximum frame size of 1KB and a maximum latency of 50ms; the QUIC protocol is used to transmit video streams and virtual asset data, with a maximum transmission unit of 1500 bytes and a maximum latency of 80ms when the maximum packet loss rate is 1%; the consistent hashing algorithm maps user requests to server nodes, and the virtual node mechanism handles request redirection when nodes change.

[0282] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A live streaming interactive system, characterized in that, include: The module includes a user-side interaction module, a broadcaster-side management module, a real-time data acquisition module, an intelligent scenario-based interaction engine, a dynamic content recommendation module, a virtual asset scenario-based generation module, a real-time communication relay module, a cloud storage module, and a backend management module. The user-end interaction module is used to provide a virtual scene entrance and a multimodal interactive interface, receive and display dynamic content and virtual assets from the real-time communication relay module, and transmit user interaction operation data to the real-time data acquisition module. The broadcaster management module is used to receive permission configuration instructions from the backend management module, manage the virtual scene and interaction strategy switch of the live broadcast room, transmit the broadcaster operation data to the real-time data acquisition module, and receive interaction strategy instructions from the intelligent scene-based interaction engine and user interaction data from the real-time communication relay module. The real-time data acquisition module is used to collect user interaction operation data transmitted by the user terminal interaction module and anchor operation data transmitted by the anchor terminal management module, store the collected data in the cloud storage module, and transmit the real-time data to the intelligent scenario-based interaction engine. The intelligent scenario-based interaction engine is used to receive real-time data transmitted by the real-time data acquisition module, generate interaction strategy instructions based on preset algorithms, and transmit the instructions to the dynamic content recommendation module, the virtual asset scenario generation module and the real-time communication relay module respectively. At the same time, it reads historical data from the cloud storage module for model optimization. The dynamic content recommendation module is used to receive the interaction strategy instructions of the intelligent scenario-based interaction engine, read the corresponding dynamic content resources from the cloud storage module, generate a personalized recommendation list, and transmit the recommendation list to the real-time communication relay module. The virtual asset scenario generation module is used to receive the interaction strategy instructions of the intelligent scenario-based interaction engine, call the preset virtual asset template and generate scenario-specific virtual assets, and transmit the generated virtual assets to the real-time communication relay module. The real-time communication relay module is used to relay data between the user terminal interaction module, the broadcaster terminal management module, the intelligent scenario-based interaction engine, the dynamic content recommendation module, and the virtual asset scenario-based generation module. It adopts a hybrid protocol to achieve low-latency transmission of different types of data. The cloud storage module is used to store real-time data transmitted by the real-time data acquisition module, historical data required by the intelligent scenario-based interactive engine, dynamic content resources of the dynamic content recommendation module, and virtual asset templates of the virtual asset scenario-based generation module, while providing a data query interface to the backend management module. The backend management module is used to review the qualifications of the anchor and configure the permissions of the live broadcast room, adjust the algorithm parameters of the intelligent scene-based interaction engine, read data from the cloud storage module to generate statistical reports, and transmit the permission configuration instructions to the anchor management module.

2. The live streaming interactive system according to claim 1, characterized in that, The user-end interaction module includes a virtual scene entry submodule and a multimodal interaction submodule. The virtual scene entry submodule provides an access point to the 3D virtual live streaming room. Users can create or select a virtual avatar and enter the virtual live streaming room through this entry point. The multimodal interaction submodule supports multimodal input of voice, gestures, and facial expressions. Users can trigger specific interactive operations through voice commands or gestures.

3. The live streaming interactive system according to claim 1, characterized in that, The intelligent scenario-based interaction engine includes a user behavior profiling submodule and a scenario matching submodule. The user behavior profiling submodule constructs a real-time dynamic profile of the user based on real-time collected multi-dimensional data. The scenario matching submodule generates an interaction strategy that is adapted to the scenario based on the real-time dynamic profile of the user and the current live streaming scenario tags.

4. The live streaming interactive system according to claim 1, characterized in that, The virtual asset scenario generation module includes an asset template library and a dynamic generation sub-module. The asset template library stores various types of virtual asset templates. The dynamic generation sub-module, based on the interaction strategy output by the intelligent scenario-based interaction engine, calls the corresponding template from the asset template library and generates scenario-based virtual assets in combination with the current live streaming scenario parameters.

5. A live streaming interactive system according to claim 1, characterized in that, The real-time communication relay module adopts a hybrid communication method using WebSocket and QUIC protocols. The WebSocket protocol is used for the transmission of text-based interactive data, while the QUIC protocol is used for the transmission of video streams and virtual scene data.

6. The live streaming interaction method according to claim 1, providing method support for the live streaming interaction system according to claims 1-5, characterized in that, Includes the following steps: S1. User initiates interaction request: The user initiates an interaction request in the virtual scene by making a heart shape with a gesture or sending a voice comment. The request carries the interaction type and real-time scene tag. S2. Real-time data acquisition and storage: The real-time data acquisition module synchronously collects the user's key gestures, voice emotion features, and the anchor's scene switching operation data, and temporarily stores the raw data in the hot data storage unit. S3. Intelligent Interaction Strategy Generation: The intelligent scenario-based interaction engine uses Attention-LSTM to extract user interests and emotional states, and then uses the DQN model to generate an interaction strategy that is adapted to the current scenario. S4. Generating Recommendations and Gifts: The dynamic content recommendation module retrieves short videos or task lists that match the scene, and the virtual asset scene generation module generates exclusive gifts with the anchor's avatar by adjusting the template parameters through GAN. S5. Low-latency push content: The real-time communication relay module uses WebSocket to push text and QUIC to transmit virtual assets, and uses a consistent hashing algorithm to ensure low-latency content reception across multiple terminals. S6. Render and Feedback Data: The virtual scene on the user end renders a unique gift animation, the host end displays statistics on the execution of the interaction strategy, and both parties send feedback data such as participation time and task completion rate back to the collection module.

7. A live streaming interaction method according to claim 6, characterized in that, The Attention-LSTM model in S3 includes an input layer, an LSTM hidden layer, an attention layer, and an output layer. The input layer receives multimodal data collected in real time. The LSTM hidden layer captures the dependencies of time-series data and outputs a hidden state sequence. The attention layer assigns higher weights to key behavioral data in the hidden state sequence. The output layer generates user interest preference vectors and emotion state values ​​for subsequent interaction strategy generation.

8. A live streaming interaction method according to claim 6, characterized in that, The DQN model in S3 employs a reinforcement learning framework, including a state space, an action space, and a reward function. The state space contains real-time dynamic user profiles, current live stream scene tags, and interaction popularity data. The action space includes operations such as sending virtual gifts, initiating interactive tasks, and recommending short videos. The reward function is composed of a weighted average of retention rate improvement, interaction frequency improvement, and virtual asset consumption growth. The DQN model stores historical interaction data through an experience replay mechanism, processes training fluctuations through target network separation, and generates the optimal interaction strategy adapted to the scenario.

9. A live streaming interaction method according to claim 6, characterized in that, The GAN model described in S4 includes a generator and a discriminator. The generator receives interaction strategy parameters and virtual asset template parameters output by the intelligent scene-based interaction engine, and generates scene-specific virtual assets by combining the anchor's avatar feature points. The discriminator compares the generated virtual assets with the real template and provides feedback on the error to optimize the generator. The virtual assets generated by the GAN model are in GLB format, support real-time rendering, and have a maximum generation time of 50ms for a single asset.

10. A live streaming interaction method according to claim 6, characterized in that, The hybrid communication method and load balancing mechanism in S5 are as follows: Text-based interactive strategy data is transmitted using the WebSocket protocol, with a maximum frame size of 1KB and a maximum latency of 50ms; video streams and virtual asset data are transmitted using the QUIC protocol, with a maximum transmission unit of 1500 bytes and a maximum latency of 80ms when the maximum packet loss rate is 1%; the consistent hashing algorithm maps user requests to server nodes, and a virtual node mechanism handles request redirection when nodes change.

Images