An augmented reality (AR) fused immersive live broadcast interaction method and device

Abstract

The present specification discloses an AR-fused immersive live broadcast interaction method and device, which can include: receiving user interaction instructions of a client, biological feature signaling and real-time video stream of a host end, wherein the biological feature signaling is generated by a local AI of the client identifying an original environment image, the interaction instructions are analyzed to obtain effective AR instructions, a target instruction is selected according to a preset arbitration mechanism and unique AR event triggering signaling is generated, 3D virtual resources are dispatched in response to the signaling, timestamped synchronous rendering instructions are generated, and the host video stream is initially AR-rendered to synthesize a mixed video stream containing initial AR content according to the instructions, biological feature signaling of multiple clients aiming at the same AR event is aggregated, group intention is obtained by statistical analysis in a set time window, AR event parameters are updated according to the group intention, and updated rendering instructions are generated, and the updated mixed video stream is dynamically rendered and secondarily synthesized, and is distributed to each client through a content distribution network.

Description

Technical Field

[0001] This invention relates to the field of online live streaming technology, and more specifically, to an immersive live streaming interactive method and apparatus that integrates AR. Background Technology

[0002] With the rapid development of 5G networks, cloud computing, and mobile internet technologies, the online live streaming industry has experienced explosive growth, becoming an important part of e-commerce, social entertainment, and online education. Current live streaming primarily relies on one-way content output from the streamer and simple text or graphic interactions from viewers via bullet comments, comments, and virtual gifts. While this interaction model establishes a connection between streamers and viewers to some extent, its interactive dimensions are limited, and its immersive experience is weak, failing to meet users' growing demands for deeper engagement and personalized experiences. Specifically, existing live streaming interactive systems suffer from the following technical problems: limited interaction formats and severe homogenization of experiences. Whether it's live streaming for entertainment, e-commerce, or games, the core interaction method remains a "bullet comment / gift" system based on text and preset animations. Bullet comments sent by users appear as scrolling text, while gifts trigger a fixed animation effect. This format lacks novelty, resulting in little difference in user experience across different live streaming rooms, easily leading to aesthetic fatigue and hindering further improvement in user engagement. Summary of the Invention

[0003] This specification provides an immersive live streaming interactive method and device that integrates AR, in order to overcome at least one technical problem existing in related technologies.

[0004] According to a first aspect of the embodiments of this specification, an immersive live streaming interactive method integrating AR is provided, applied on a server side, comprising:

[0005] The system receives user interaction command data, user biometric signaling, and real-time video stream from at least one client; wherein the user biometric signaling is structured data generated by the client through local AI recognition of the collected raw environmental images.

[0006] The user interaction command data is parsed to obtain valid AR interaction commands;

[0007] According to the preset arbitration mechanism, a target AR interaction instruction is selected from the valid AR interaction instructions, and a unique AR event triggering signal is generated based on the target AR interaction instruction;

[0008] In response to the AR event trigger signal, the corresponding 3D virtual resources are scheduled from the pre-set resource database, and a synchronous rendering instruction with a timestamp is generated;

[0009] Based on the synchronous rendering instructions, the real-time video stream of the broadcaster is subjected to initial AR rendering and image synthesis to generate an augmented reality hybrid video stream containing initial AR content.

[0010] Receive and aggregate user biometric signaling from multiple clients for the same AR event, and perform statistical analysis on the user biometric signaling for the same AR event within a set time window to obtain the group intent analysis results;

[0011] Update the state parameters of the AR event based on the group intent analysis results, and generate corresponding update rendering instructions;

[0012] Based on the update rendering instruction, dynamic AR rendering and secondary synthesis are performed on the real-time video stream of the broadcaster and the initial AR content to update the augmented reality hybrid video stream;

[0013] The updated augmented reality hybrid video stream is distributed to each client via a content delivery network.

[0014] In some optional implementations, parsing the user interaction command data to obtain valid AR interaction commands includes:

[0015] The bullet screen text in the received user interaction command data is cleaned and segmented to filter out invalid information and obtain the cleaned bullet screen text.

[0016] The AC automaton algorithm is used to match preset keywords in the cleaned bullet screen text, and a lightweight text classification model is used to identify the potential instruction intent in the cleaned bullet screen text.

[0017] Based on the matching results of the preset keywords and the recognition results of the potential instruction intent, a valid AR interaction instruction that meets the preset conditions is obtained.

[0018] In some optional implementations, selecting a target AR interaction command from the valid AR interaction commands according to a preset arbitration mechanism includes:

[0019] A weighted scoring mechanism based on time windows is adopted to calculate the weighted score for each valid AR interaction command;

[0020] Within the time window, the valid AR interaction instruction with the highest weighted score is selected as the target AR interaction instruction;

[0021] The weighted scoring factors include the user level that sent the valid AR interaction command, the consumption value associated with the valid AR interaction command, the user's historical interaction frequency, and the time decay factor.

[0022] In some optional implementations, the step of receiving and aggregating user biometric signaling from multiple clients for the same AR event, and performing statistical analysis on the user biometric signaling for the same AR event within a set time window to obtain group intent analysis results includes:

[0023] Within the set time window, the quantity and type of various user biometric signaling related to the same AR event are counted to obtain statistical results;

[0024] Based on the statistical results, the proportion, frequency of occurrence, and popularity trend of various interactive behaviors are calculated.

[0025] The analysis results representing the highest-level group interaction intentions are output as the group intention analysis results.

[0026] In some optional implementations, the step of performing initial AR rendering and image compositing on the real-time video stream of the broadcaster based on the synchronous rendering instructions to generate an augmented reality hybrid video stream containing initial AR content includes:

[0027] Load the corresponding 3D virtual resources according to the synchronous rendering instructions;

[0028] The real-time video stream from the broadcaster's terminal is subjected to environmental understanding, and the physical plane of the environment where the broadcaster is located is identified through synchronous positioning and mapping or plane detection.

[0029] Analyze the lighting information of the real-time video stream, including the direction of the main light source, ambient light intensity, and color;

[0030] Based on the timestamp and position parameters in the synchronous rendering instruction, the 3D virtual resources are rendered on the physical plane, and depth testing is performed to realize the occlusion relationship between virtual objects and the real environment.

[0031] The rendered AR visual layer is pixel-level aligned and alpha-blended with the real-time video stream from the broadcaster's end to generate the augmented reality hybrid video stream containing the initial AR content.

[0032] According to a second aspect of the present invention, an immersive live streaming interactive method integrating AR is provided, applied to a client, comprising:

[0033] Collect user interaction command data, raw environmental images, and real-time video stream from the broadcaster's end;

[0034] The original environmental image is identified by a local AI engine to generate user biometric signaling, which is structured data.

[0035] The user interaction command data and the user biometric signaling are uploaded to the server.

[0036] Receive and play augmented reality hybrid video streams from the server; the augmented reality hybrid video streams are generated by the server based on the user interaction command data and user biometric signaling, after performing AR rendering and image synthesis on the real-time video stream from the broadcaster.

[0037] The augmented reality hybrid video stream includes AR content that is dynamically updated based on the results of group intent analysis. The results of group intent analysis are obtained by the server aggregating and statistically analyzing the biometric signaling of multiple clients for the same AR event.

[0038] According to a third aspect of the present invention, an immersive live streaming interactive method integrating AR is provided, applied to a network system including a client and a server, comprising:

[0039] The client collects user-input interaction command data and raw environmental images obtained through the camera. It uses a local AI engine to recognize the raw environmental images, generates structured user biometric signaling, and uploads the interaction command data and user biometric signaling to the server. At the same time, the client sends a playback request to the server for the live video stream from the broadcaster.

[0040] The server receives user interaction command data, user biometric signaling, and real-time video stream uploaded by the broadcaster from the client; it parses the user interaction command data, identifies valid AR interaction commands, selects the target AR interaction command from them according to a preset arbitration mechanism, and generates a unique AR event triggering signal.

[0041] In response to the AR event triggering signal, the server schedules the corresponding 3D virtual resources from the resource database, generates a timestamped synchronous rendering instruction, and performs initial AR rendering and image synthesis on the real-time video stream of the broadcaster based on the instruction, generating an augmented reality hybrid video stream containing initial AR content.

[0042] The server receives and aggregates user biometric signaling from multiple clients for the same AR event, performs statistical analysis within a set time window to obtain the group intent analysis results, updates the state parameters of the AR event based on the results, generates update rendering instructions, and performs dynamic AR rendering and secondary compositing on the real-time video stream of the broadcaster and the rendered initial AR content accordingly, updates the augmented reality hybrid video stream, and finally distributes the updated video stream to each client through the content delivery network.

[0043] The client receives and decodes the augmented reality hybrid video stream from the server; in scenarios requiring personalized interaction, it receives independent AR rendering instructions from the server and overlays personalized interactive elements visible only to the current user onto the locally played hybrid video stream.

[0044] According to a fourth aspect of the present invention, an immersive live streaming interactive device integrating AR is provided, applied on a server side, comprising:

[0045] The data receiving and access module is used to receive user interaction command data, user biometric signaling, and real-time video stream from at least one client; wherein, the user biometric signaling is structured data generated by the client through local AI recognition of the collected raw environmental images;

[0046] The instruction parsing module is used to parse the user interaction instruction data to obtain valid AR interaction instructions;

[0047] The event-triggered arbitration module is used to select a target AR interaction instruction from the valid AR interaction instructions according to a preset arbitration mechanism, and generate a unique AR event triggering signal based on the target AR interaction instruction;

[0048] The AR event management and resource scheduling module is used to respond to the AR event triggering signal, schedule the corresponding 3D virtual resources from the pre-set resource database, and generate a synchronous rendering instruction with a timestamp.

[0049] The initial AR rendering and compositing module is used to perform initial AR rendering and image compositing on the real-time video stream of the broadcaster based on the synchronous rendering instructions, and generate an augmented reality hybrid video stream containing initial AR content.

[0050] The user interaction analysis and group intent recognition module is used to receive and aggregate user biometric signaling from multiple clients for the same AR event, and perform statistical analysis on the user biometric signaling for the same AR event within a set time window to obtain the group intent analysis results.

[0051] The AR event status update and instruction generation module is used to update the status parameters of the AR event based on the group intent analysis results and generate corresponding update rendering instructions.

[0052] The dynamic AR rendering and secondary compositing module is used to perform dynamic AR rendering and secondary compositing on the real-time video stream of the broadcaster and the initial AR content based on the update rendering command, so as to update the augmented reality hybrid video stream.

[0053] The streaming media distribution module is used to distribute updated augmented reality hybrid video streams to various clients via a content delivery network.

[0054] According to a fifth aspect of the present invention, an immersive live streaming interactive device integrating AR is provided, applied to a client, comprising:

[0055] The data acquisition module is used to collect user interaction command data, raw environmental images, and real-time video streams from the broadcaster's end.

[0056] The local AI recognition and feature generation module is used to recognize the original environmental image through a local AI engine and generate user biometric signaling, wherein the user biometric signaling is structured data.

[0057] The data upload module is used to upload the user interaction command data and the user biometric signaling to the server.

[0058] The hybrid video stream receiving and playback module is used to receive and play the augmented reality hybrid video stream from the server. The augmented reality hybrid video stream is generated by the server based on the user interaction command data and user biometric signaling, after performing AR rendering and image synthesis on the real-time video stream from the broadcaster.

[0059] The augmented reality hybrid video stream includes AR content that is dynamically updated based on the results of group intent analysis. The results of group intent analysis are obtained by the server aggregating and statistically analyzing the biometric signaling of multiple clients for the same AR event.

[0060] The beneficial effects of the embodiments in this specification are as follows:

[0061] This technical solution organically integrates AR rendering technology, real-time user behavior analysis, and live streaming interaction mechanisms, enabling users to transform from passive viewers to active participants. Users can summon virtual 3D items in the live stream by sending specific bullet comments or making payments, and interact using gestures, significantly enhancing their sense of participation and control. Simultaneously, by aggregating the gestures and facial expressions of multiple users in real time, the collective emotions or intentions of the group can be transformed into grand AR visual effects, enhancing the sense of presence and community in the live stream and effectively stimulating collective consumption behavior. Furthermore, the introduction of AR technology makes various interactive scenarios more fun, ritualistic, and transparent, helping to extend user dwell time and increase willingness to pay. It also generates lightweight biometric signals through local AI recognition on the client side and uploads them, reducing the uplink bandwidth pressure and computational load on the server. Combined with command arbitration and interaction aggregation mechanisms, it supports a massive number of users participating online simultaneously while ensuring interactive effects, exhibiting good scalability and stability, elevating live streaming interaction from a two-dimensional information level to a three-dimensional immersive experience. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments or related technologies of this specification, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 A flowchart illustrating an immersive live streaming interactive method incorporating AR, as provided in this application;

[0064] Figure 2 The corresponding to this application Figure 1 A structural diagram of an immersive live-streaming interactive device that integrates AR;

[0065] Figure 3 This is a flowchart illustrating the interaction process between the client and server modules in an immersive live streaming interactive system that integrates AR. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of one or more embodiments of this specification clearer, the technical solutions of one or more embodiments of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of one or more embodiments of this specification.

[0067] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another.

[0068] The technical solution of this application will be described below with reference to the accompanying drawings. Figure 1 This specification provides a flowchart illustrating an immersive live streaming interaction method integrating AR, applied to the server side. The following is based on... Figure 1 The technical solution of this application is described, and the method may include the following steps:

[0069] Step 102: Receive user interaction command data, user biometric signaling, and real-time video stream from at least one client; wherein the user biometric signaling is structured data generated by the client through local AI recognition of the collected raw environmental images.

[0070] In this step, the user interaction command data comes from the input of the audience on the client side. Specifically, it can include the bullet screen text sent by the audience, the behavior record of sending virtual gifts, etc. This data is the command carrier for the audience to interact with the live broadcast scene. It is collected by the client and uploaded to the server.

[0071] User biometric signaling also originates from the client side. Its generation process can be as follows: The client captures raw environmental images of the viewer via a camera. These raw images may contain biometric features such as user gestures and facial expressions. The client's local AI engine, such as lightweight computer vision models like MobileNetV3 or YOLOv5-lite, then processes the raw images to generate structured data. This structured data does not contain the raw video stream; it only presents the recognition results in a structured form, such as a packet containing user identifiers, gesture categories (e.g., 'like', 'open palm'), timestamps, and recognition confidence scores. The real-time video stream from the broadcaster's device is acquired by the broadcaster's capture device and uploaded to the server. It typically contains the broadcaster's real-time image, encoded in 1080p, 60fps H.265, etc., and serves as the base image for AR content synthesis. The server aggregates these three types of data, providing basic input for subsequent interactive parsing, AR event triggering, and other processes.

[0072] Step 104: Parse the user interaction command data to obtain valid AR interaction commands.

[0073] The purpose of this step is to parse user interaction command data, such as processing interactive commands like bullet screen text sent by viewers. During processing, this raw data can first be preprocessed to filter out meaningless emoticons, spam comments, and other invalid information. The remaining text is then segmented into word segments to create structured content suitable for further analysis. Afterward, the processed text can be analyzed using preset rules and natural language processing techniques. This involves matching preset keywords, such as specific command words related to AR interaction, and determining the emotional tone and potential interactive intent within the text, such as identifying the user's desire to trigger a certain type of AR event.

[0074] In this step, through such a parsing process, instructions that meet preset conditions and can actually drive AR interaction can be filtered from the user interaction instruction data, thus obtaining valid AR interaction instructions.

[0075] Step 106: Based on the preset arbitration mechanism, select a target AR interaction instruction from the valid AR interaction instructions, and generate a unique AR event triggering signaling based on the target AR interaction instruction.

[0076] The main purpose of this step is to select the target AR interaction command from multiple valid AR interaction commands. That is, when multiple valid AR interaction commands exist at the same time, the commands are processed within a set time window, such as hundreds of milliseconds, through a preset arbitration mechanism to select the target AR interaction command.

[0077] The arbitration mechanism employs a weighted scoring method to evaluate each valid instruction. The scoring considers factors such as the user's level, the associated consumption value, the user's historical interaction frequency, and the instruction's waiting time in the queue. This is achieved through a time decay factor, prioritizing new instructions. At the end of each time window, the highest-scoring valid AR interaction instruction is selected as the target AR interaction instruction.

[0078] This step then generates an AR event trigger signaling based on the selected target AR interaction command. This signaling is unique and can carry a unique event identifier, such as an eventId in a specific format, to ensure that the subsequent creation, management, and synchronization of AR events can accurately correspond and avoid confusion.

[0079] Step 108: In response to the AR event trigger signal, schedule the corresponding 3D virtual resources from the pre-set resource database and generate a timestamped synchronous rendering instruction.

[0080] In this step, upon receiving an AR event trigger signal, the corresponding processing flow will be initiated. First, the module responsible for managing AR events on the server side can respond to this signal. Based on the event type and related parameters contained in the signal, it retrieves the corresponding 3D virtual resources from a pre-established resource database. These resources may include, but are not limited to, 3D models, matching materials, preset animation sequences, and associated sound effects to meet the presentation requirements of the AR event.

[0081] While scheduling resources, a synchronization rendering instruction is generated. This instruction can carry a precise timestamp, which can be generated based on the Network Time Protocol (NTP). This timestamp explicitly indicates on which video frame of the broadcaster's live video stream the AR virtual resource should begin to be presented. This time-stamped synchronization mechanism in this step solves the problem of asynchronous AR content display between different clients due to network latency and other factors, ensuring that all viewers see AR events in a consistent time frame.

[0082] In addition, the synchronous rendering instruction can also include parameters such as the initial position and scaling ratio of the AR virtual resources, thereby providing a clear execution basis for the subsequent rendering and compositing process.

[0083] Step 110: Based on the synchronous rendering instruction, perform initial AR rendering and image synthesis on the real-time video stream of the broadcaster to generate an augmented reality hybrid video stream containing initial AR content.

[0084] In this step, when performing initial AR rendering and image compositing on the live video stream from the broadcaster's end based on synchronous rendering commands, the corresponding 3D virtual resources can be loaded first. These resources can include 3D models, materials, animations, etc. Next, the broadcaster's live video stream is processed to understand its environment. Through synchronous positioning and mapping or plane detection, physical planes in the real-world environment surrounding the broadcaster, such as a desktop or the ground, are identified, providing spatial references for the placement of the 3D virtual resources. Simultaneously, the lighting conditions in the video stream can be analyzed to determine the direction of the main light source, the intensity and color of the ambient light, ensuring that the lighting effects of the 3D virtual resources match the real environment.

[0085] Next, this step can render 3D virtual resources on the identified physical plane according to the timestamp and position parameters in the synchronous rendering instruction. During this process, depth testing can be performed to handle the occlusion relationship between virtual objects and the anchor or other objects in the real environment, so that the virtual and real images can be naturally integrated.

[0086] Finally, the rendered AR visual layer is pixel-level alpha blended with the real-time video stream from the broadcaster's end to obtain an augmented reality hybrid video stream containing the initial AR content.

[0087] Step 112: Receive and aggregate user biometric signaling from multiple clients for the same AR event, and perform statistical analysis on the user biometric signaling for the same AR event within a set time window to obtain the group intent analysis results.

[0088] In this step, user biometric signals will be continuously received from multiple clients. These signals all correspond to the same AR event. As described in step 102, the user biometric signal is structured data generated by the client through local AI recognition technology after processing the collected original environmental images such as gestures and expressions. It is not the original video stream.

[0089] Then, within a defined time window, such as a very short period of a few hundred milliseconds, these signaling messages for the same AR event can be aggregated. Subsequently, within this time window, statistical analysis is performed on the aggregated signaling messages. This involves first identifying the specific type of these signaling messages, such as a thumbs-up gesture, an open palm gesture, or a certain facial expression, and then counting the number of each type of signaling message. Based on this basic type and quantity information, the proportion of each type of interaction behavior in the overall picture, its frequency of occurrence, and the trend of popularity changes can be calculated, such as the growth of a certain interaction behavior in a short period of time.

[0090] Through such statistical analysis, the final result of the group intention analysis can be formed. This result is a comprehensive judgment on the group's interaction tendency in the AR event. For example, it can reflect what type of interaction most users are doing, or whether the popularity of a certain interaction is rising, and clearly present the group's overall interaction intention.

[0091] Step 114: Update the state parameters of the AR event based on the group intent analysis results, and generate the corresponding update rendering instructions.

[0092] This step involves adjusting the state parameters of the AR event based on the results of the group intent analysis. These results, such as the popularity of group interaction, the proportion or trend of various interactive behaviors, directly affect the current state of the AR event. For example, when the analysis shows that the popularity of the group's participation in the "gathering energy" interaction reaches a certain level, the state parameters of the virtual object in the AR event, such as the light effect intensity and animation progress, can be adjusted accordingly. If it involves an auction-type AR event, the current price, countdown, and other parameters can be updated based on the activity level of the group interaction.

[0093] In this step, while updating the state parameters, a corresponding update rendering instruction will be generated. This instruction will clearly record the specific parameters that need to be adjusted, such as the intensity value of the light effect, the stage marker of the animation, the magnitude of the change in the value, etc., and carry a synchronization timestamp to ensure that the subsequent rendering process can be executed accurately according to this instruction, so that the state changes of the AR event are consistent with the rhythm of the group interaction and can be presented to all clients synchronously.

[0094] Step 116: Based on the update rendering instruction, perform dynamic AR rendering and secondary synthesis on the real-time video stream of the broadcaster and the initial AR content to update the augmented reality hybrid video stream.

[0095] In this step, when dynamically rendering and compositing the real-time video stream and initial AR content on the broadcaster's end based on the update rendering instructions, the existing initial AR content can be dynamically updated first according to the adjustment requirements specified in the instructions. For example, if the instructions require enhancing the lighting effects of virtual objects, the brightness, particle density, and other parameters of the lighting effects will be adjusted based on the level of group interaction. If animation changes are involved, the animation stages of virtual objects can be advanced or switched, such as the transition of a card pack from "energy gathering" to "opening".

[0096] At the same time, this step can combine the current frame state of the real-time video stream on the broadcaster's end to reprocess the adaptation relationship between virtual content and real scene. This includes fine-tuning the lighting effects of virtual objects according to the lighting changes in the real-time video, and using depth detection to ensure that virtual objects and dynamically changing objects in the broadcaster or real environment, such as the broadcaster's gestures, maintain a natural occlusion relationship to avoid any glitches.

[0097] Next, the dynamically updated AR content is combined with the real-time video stream from the broadcaster's end at a second pixel level. Alpha blending and other methods are used to seamlessly integrate the updated virtual content with the real scene, ultimately generating an updated augmented reality hybrid video stream to ensure that the AR content seen by all viewers is synchronized with the progress of group interaction.

[0098] Step 118: Distribute the updated augmented reality hybrid video stream to each client via the content delivery network.

[0099] In this step, the updated augmented reality hybrid video stream is first encoded using an encoding format suitable for streaming media transmission, such as H.264 or H.265, and then encapsulated into a format that conforms to network transmission standards, such as HLS or DASH, so as to facilitate efficient transmission over the network.

[0100] Subsequently, these processed video streams can be distributed through a content delivery network. During the distribution process, the video bitrate can be dynamically adjusted according to the network conditions of different clients. For example, a lower bitrate video stream can be pushed to clients with low network bandwidth, while a higher definition video stream can be pushed to clients with better network conditions, so as to ensure that clients in different environments can receive the video smoothly.

[0101] This technical solution organically integrates AR rendering technology, real-time user behavior analysis, and live streaming interaction mechanisms, enabling users to transform from passive viewers to active participants. Users can summon virtual 3D items in the live stream by sending specific bullet comments or making payments, and interact using gestures, significantly enhancing their sense of participation and control. Simultaneously, by aggregating the gestures and facial expressions of multiple users in real time, the collective emotions or intentions of the group can be transformed into grand AR visual effects, enhancing the sense of presence and community in the live stream and effectively stimulating collective consumption behavior. Furthermore, the introduction of AR technology makes various interactive scenarios more fun, ritualistic, and transparent, helping to extend user dwell time and increase willingness to pay. This technical solution also generates lightweight biometric signals through local AI recognition on the client side and uploads them, reducing the uplink bandwidth pressure and computational load on the server. Combined with command arbitration and interaction aggregation mechanisms, it supports a massive number of users participating online simultaneously while ensuring interactive effects, exhibiting good scalability and stability, elevating live streaming interaction from a two-dimensional information level to a three-dimensional immersive experience.

[0102] Based on the technical solutions described above, this application also provides some more specific technical solutions, which are described below.

[0103] In an optional embodiment, parsing the user interaction command data to obtain valid AR interaction commands may include:

[0104] The bullet screen text in the received user interaction command data is cleaned and segmented to filter out invalid information and obtain the cleaned bullet screen text.

[0105] The AC automaton algorithm is used to match preset keywords in the cleaned bullet screen text, and a lightweight text classification model is used to identify the potential instruction intent in the cleaned bullet screen text.

[0106] Based on the matching results of the preset keywords and the recognition results of the potential instruction intent, a valid AR interaction instruction that meets the preset conditions is obtained.

[0107] In this embodiment, when parsing user interaction command data to obtain valid AR interaction commands, the bullet screen text in the received user interaction command data can first be processed. Meaningless emoticons, spam comments, and other invalid information are removed through cleaning, followed by word segmentation to obtain cleaned bullet screen text. Then, algorithms such as the AC automata can be used to match preset keywords to the cleaned bullet screen text. Simultaneously, a lightweight text classification model can be combined to identify potential command intentions in the text, such as determining whether the user has a need to trigger an AR event. Finally, based on the matching results of preset keywords and the identification results of potential command intentions, commands that meet preset conditions are selected; these commands are the valid AR interaction commands.

[0108] In an optional embodiment, selecting a target AR interaction command from the valid AR interaction commands based on a preset arbitration mechanism may include:

[0109] A weighted scoring mechanism based on time windows is adopted to calculate the weighted score for each valid AR interaction command;

[0110] Within the time window, the valid AR interaction instruction with the highest weighted score is selected as the target AR interaction instruction;

[0111] The weighted scoring factors include the user level that sent the valid AR interaction command, the consumption value associated with the valid AR interaction command, the user's historical interaction frequency, and the time decay factor.

[0112] In this embodiment, when selecting a target AR interaction command from valid AR interaction commands based on a preset arbitration mechanism, a weighted scoring method based on a time window can be adopted. Specifically, a weighted score can be calculated for each valid AR interaction command within a set time window. When calculating the score, factors such as the user level sending the command, the associated consumption value of the command, the sending user's historical interaction frequency, and a time decay factor can be comprehensively considered. Considering the time decay factor ensures that the longer the command waits in the queue, the greater the impact on the score, thus making new commands more noticeable.

[0113] Then, in this embodiment, at the end of this time window, the one with the highest weighted score can be selected from all valid AR interaction commands and used as the target AR interaction command. In this way, when multiple valid commands exist at the same time, the priority can be reasonably determined to ensure that the triggering of AR events is orderly and meets expectations.

[0114] In an optional embodiment, the step of receiving and aggregating user biometric signaling from multiple clients for the same AR event, and performing statistical analysis on the user biometric signaling for the same AR event within a set time window to obtain group intent analysis results, may include:

[0115] Within the set time window, the quantity and type of various user biometric signaling related to the same AR event are counted to obtain statistical results;

[0116] Based on the statistical results, the proportion, frequency of occurrence, and popularity trend of various interactive behaviors are calculated.

[0117] The analysis results representing the highest-level group interaction intentions are output as the group intention analysis results.

[0118] In this embodiment, during the process of receiving and aggregating user biometric signaling from multiple clients for the same AR event, and performing statistical analysis on these signaling within a set time window to obtain group intent analysis results, all user biometric signaling related to the AR event can be collected within the set time window. This signaling is structured data generated by the client through local AI recognition of raw environmental images, such as gestures and expressions. Within this time window, the collected signaling can be sorted out, and the quantity and specific type of different types of signaling can be statistically determined to form basic statistical results.

[0119] Subsequently, based on these statistical results, we can further calculate the proportion of various interactive behaviors in the whole, their frequency of occurrence, and the trend of popularity, such as the increase or decrease of a certain type of interaction in a short period of time.

[0120] Ultimately, through such statistics and calculations, the output analysis results will reflect the highest level of interaction intent of the group in the AR event, that is, the extraction and summary of the group's overall interaction tendency.

[0121] In an optional embodiment, the step of performing initial AR rendering and image synthesis on the real-time video stream of the broadcaster based on the synchronous rendering command to generate an augmented reality hybrid video stream containing initial AR content may include:

[0122] Load the corresponding 3D virtual resources according to the synchronous rendering instructions;

[0123] The real-time video stream from the broadcaster's terminal is subjected to environmental understanding, and the physical plane of the environment where the broadcaster is located is identified through synchronous positioning and mapping or plane detection.

[0124] Analyze the lighting information of the real-time video stream, including the direction of the main light source, ambient light intensity, and color;

[0125] Based on the timestamp and position parameters in the synchronous rendering instruction, the 3D virtual resources are rendered on the physical plane, and depth testing is performed to realize the occlusion relationship between virtual objects and the real environment.

[0126] The rendered AR visual layer is pixel-level aligned and alpha-blended with the real-time video stream from the broadcaster's end to generate the augmented reality hybrid video stream containing the initial AR content.

[0127] In this step, the corresponding 3D virtual resources can be loaded first according to the content specified in the synchronous rendering instructions. These resources can include 3D models, adapted materials, preset animation sequences, and associated sound effects to meet the initial presentation requirements of the AR event.

[0128] Next, environmental understanding processing is performed on the real-time video stream from the broadcaster's end. Through synchronous positioning and mapping technology or plane detection methods, the physical planes in the real environment where the broadcaster is located, such as the desktop and the ground, are identified, providing a spatial basis that fits the real scene for the subsequent placement of 3D virtual resources.

[0129] Simultaneously, the lighting information in the real-time video stream is analyzed, specifically including the direction of the main light source, the intensity of ambient light, and the color of ambient light. By acquiring these lighting parameters, the subsequently rendered 3D virtual resources can simulate lighting and shadow effects consistent with the real environment, enhancing the integration of virtual objects into the real scene. Then, based on the timestamp and position parameters carried in the synchronous rendering command, the loaded 3D virtual resources are accurately rendered onto the previously identified physical plane. During the rendering process, depth testing technology is used to handle the occlusion relationships between virtual objects and the broadcaster, or other objects in the real environment, ensuring that virtual objects present a sense of depth and perspective consistent with real spatial logic in the image.

[0130] Finally, the rendered AR visual layer is precisely aligned pixel-level with the real-time video stream on the broadcaster's end, and then alpha blending technology is used to merge the two, so that the virtual content and the real video screen are seamlessly connected, ultimately generating an augmented reality hybrid video stream containing the initial AR content.

[0131] According to the above explanation, such as Figure 2 As shown in the embodiments of this specification, a signaling interaction diagram of an immersive live streaming interaction method integrating AR in a practical application scenario is also provided, including:

[0132] Step 202: The user initiates an AR interactive operation request through the client.

[0133] As mentioned earlier, users can trigger interaction requests by sending bullet comments, gift rewards, or body gestures. The client's interaction acquisition component captures this multimodal data in real time. The acquisition process follows the audio and video synchronization standards of AR live streaming systems, using camera array synchronization technology to align the timestamps of facial expression images, voice signals, and body movements, ensuring the spatiotemporal consistency of the raw data. This data will serve as the basic input for AR event generation. Analogous to the preprocessing mechanism in multi-view video capture, data quality will be initially optimized through operations such as noise reduction and format conversion.

[0134] Step 204: The client-side data acquisition and preprocessing module extracts features from the raw data.

[0135] In this step, the client employs a lightweight AI processing framework to perform facial landmark detection and alignment on the collected image data, and Mel-spectrum conversion on the speech signal to extract emotional features such as tone and speech rate. The processed structured data includes user identification, interaction type, and feature strength values. It is encapsulated according to streaming media protocols through an encrypted transmission channel, and uses the same compression encoding strategy as AR video streams to reduce bandwidth consumption and ensure that data transmission latency is controlled within a preset threshold.

[0136] Step 206: The client uploads the preprocessed data to the server-side instruction parsing module.

[0137] The data packet header contains the terminal device identifier and timestamp information, which is consistent with the AR system's device identification mechanism, facilitating subsequent data traceability and synchronization. The transmission process can use the TLS encryption protocol to ensure data security. For high-priority interactive requests, such as AR effects triggered by gifts, priority transmission can be achieved by setting the priority field in the data packet header.

[0138] Step 208: The instruction parsing and event triggering module converts the request into standardized AR event instructions.

[0139] In this step, the server performs NLP analysis and instruction arbitration on the received interactive data, converting the user's natural language requests or behavioral instructions into executable AR event types, such as virtual prop generation and scene switching. The instruction data may include the target AR resource identifier, triggering conditions, and execution priority. The resource identifier format follows the interaction specifications with the resource database to ensure the accuracy of subsequent resource calls.

[0140] Step 210: The AR event management and synchronization module coordinates resource loading and state initialization.

[0141] After instruction parsing is complete, the event management module initiates a query request to the resource database to obtain the corresponding 3D models, textures, and other AR resources, while simultaneously performing resource integrity verification and version matching. During initialization, spatial alignment parameters between virtual objects and the real-world scene must be calculated. Environmental feature point data generated using SLAM technology ensures stable anchoring of AR content within the live streaming scene. This process is analogous to the spatial synchronization mechanism in multi-view video capture, using timestamp calibration to ensure temporal consistency between resource loading and the video stream.

[0142] Step 212: The user interaction analysis module generates feedback on group behavior characteristics.

[0143] In this step, the module aggregates interaction data from all users in the current room, employing the same statistical methods as AR user behavior analysis to uncover patterns in group interactions, such as high-frequency interaction areas and preferred effect types. The analysis results are pushed to the AR event management module in structured data format to dynamically adjust the presentation strategy of AR content. For example, when it is detected that most users are paying attention to a certain area, the rendering priority of AR effects in that area is automatically increased.

[0144] Step 214: The server-side AR rendering and compositing module generates an augmented reality video stream.

[0145] Upon receiving the synchronized rendering command from the event management module, the rendering engine performs environmental understanding and lighting estimation based on the real-time video stream, adjusting the lighting and shadow effects of virtual objects to match the real scene. Depth testing technology handles the occlusion relationship between virtual objects and the real-world image, and a multi-resolution fusion algorithm achieves seamless stitching of AR content and the live stream. The synthesized video stream is then divided into timestamped TS slices according to the HLS protocol, awaiting distribution and transmission.

[0146] Step 216: The streaming media distribution module pushes the AR-enhanced video stream to the client.

[0147] The distribution module dynamically adjusts the video bitrate based on the client's network conditions, employs the RTMP protocol for low-latency push / pull streaming, and accelerates content distribution through CDN nodes. For live-streaming scenarios, the module integrates the broadcaster's stream and the audience's AR interaction stream, using stream synchronization technology to ensure frame-level alignment of multiple video sources and avoid video stuttering or audio-visual misalignment. The video stream received by the client includes basic video data and AR interaction metadata, supporting local secondary rendering optimization.

[0148] Step 218: The client-side playback and rendering unit presents the AR interactive effects and provides feedback data.

[0149] After the client decodes the video stream, it performs final rendering optimization of the AR content based on local device performance, ensuring smoothness by adjusting parameters such as rendering resolution and polygon count. Simultaneously, it collects real-time user feedback data on the AR effects, such as gaze focus and interaction completion rate, encapsulates it in a preset format, and uploads it to the server. This feedback data will be used to optimize the accuracy of subsequent AR event triggering, forming a closed-loop iterative mechanism of "interaction-analysis-optimization," analogous to how AR systems continuously improve rendering effects through user behavior data.

[0150] Step 120: The server performs correlation analysis and archives the interactive data and rendering effects.

[0151] This invention also provides an immersive live streaming interaction method integrating AR, applied to a client, comprising:

[0152] It collects user interaction command data, raw environmental images, and real-time video streams from the broadcaster's end.

[0153] The original environmental image is identified by a local AI engine to generate user biometric signaling, which is structured data.

[0154] The user interaction command data and the user biometric signaling are uploaded to the server.

[0155] Receive and play augmented reality hybrid video streams from the server; the augmented reality hybrid video streams are generated by the server based on the user interaction command data and user biometric signaling, after performing AR rendering and image synthesis on the real-time video stream from the broadcaster.

[0156] The augmented reality hybrid video stream includes AR content that is dynamically updated based on the results of group intent analysis. The results of group intent analysis are obtained by the server aggregating and statistically analyzing the biometric signaling of multiple clients for the same AR event.

[0157] This invention also provides an immersive live streaming interaction method integrating AR, applied to a network system including a client and a server, comprising the following steps:

[0158] The client collects user-input interaction command data and raw environmental images obtained through the camera. It uses a local AI engine to recognize the raw environmental images, generates structured user biometric signaling, and uploads the interaction command data and user biometric signaling to the server. At the same time, the client sends a playback request to the server for the live video stream from the broadcaster.

[0159] The server receives user interaction command data, user biometric signaling, and real-time video stream uploaded by the broadcaster from the client; it parses the user interaction command data, identifies valid AR interaction commands, selects the target AR interaction command from them according to a preset arbitration mechanism, and generates a unique AR event triggering signal.

[0160] In response to the AR event triggering signal, the server schedules the corresponding 3D virtual resources from the resource database, generates a timestamped synchronous rendering instruction, and performs initial AR rendering and image synthesis on the real-time video stream of the broadcaster based on the instruction, generating an augmented reality hybrid video stream containing initial AR content.

[0161] The server receives and aggregates user biometric signaling from multiple clients for the same AR event, performs statistical analysis within a set time window to obtain the group intent analysis results, updates the state parameters of the AR event based on the results, generates update rendering instructions, and performs dynamic AR rendering and secondary compositing on the real-time video stream of the broadcaster and the rendered initial AR content accordingly, updates the augmented reality hybrid video stream, and finally distributes the updated video stream to each client through the content delivery network.

[0162] The client receives and decodes the augmented reality hybrid video stream from the server; in scenarios requiring personalized interaction, it receives independent AR rendering instructions from the server and overlays personalized interactive elements visible only to the current user onto the locally played hybrid video stream.

[0163] Example 1: System Overall Architecture and Workflow

[0164] This embodiment provides an immersive live interactive system architecture that integrates AR, mainly composed of two parts: a client and a server. The two work together through the network to achieve a multi-dimensional, real-time, and immersive live interactive experience.

[0165] The client is typically a live streaming application installed on a user's smartphone, tablet, or PC. Its core functionality is achieved through the coordinated efforts of several key components: First, it provides an interactive interface for users to watch the live stream, input bullet comments, and send gifts, offering users an entry point for participating in live interaction; second, it utilizes device hardware resources to capture high-definition video streams (e.g., H.265 encoding at 1080p resolution and 60fps) and audio streams (e.g., AAC encoding) from the broadcaster's end, while simultaneously capturing bullet comment text input from the viewer's end and video frames captured by the camera; third, it incorporates a pre-trained lightweight computer vision model (such as MobileNetV3, YOLOv5-lite, etc.) as a local AI processing core, enabling real-time processing of video frames captured by the camera on the user's device, and recognizing... It can distinguish predefined gestures (such as liking, opening palm, clenching fist, waving) or expressions (such as smiling, surprised), and does not require uploading the original video. The recognition results are packaged into a lightweight data package (such as a JSON object: {"userId":"12345","gesture":"THUMBS_UP","timestamp_ms":1663764892123,"confidence":0.92}), which saves uplink bandwidth and protects user privacy. Fourth, it can receive the final mixed video stream distributed from the server and decode and play it. In some scenarios that require local rendering on the client side (such as displaying interactive buttons that are only visible to the user), it can also receive independent AR rendering instructions and call the device's graphics interface (such as OpenGL ES, Metal) to overlay and render 2D / 3D elements on top of the video layer.

[0166] The server, acting as the system's brain, is deployed on a high-performance cloud computing platform and consists of a series of collaborative microservices. Physically, it can be a rack-mounted server cluster equipped with high-performance CPUs (such as Intel Xeon Platinum series) and professional-grade GPUs (such as NVIDIA A100 or RTX A6000). Its core functions may include:

[0167] As a traffic entry point, it receives RTMP / SRT push streams from the broadcaster and performs preprocessing such as transcoding, recording, and screenshotting; at the same time, it receives massive amounts of bullet screen data and lightweight interactive signaling from all clients through long connection protocols such as WebSocket or HTTP / 2.

[0168] The system processes the massive influx of bullet comments by first filtering out meaningless emoticons and spam. Then, it uses an efficient AC automaton algorithm to match preset keywords (such as "card unpacking," "linking," and "auctioning"). Simultaneously, a lightweight text classification model is used to determine the sentiment and potential instructions within the bullet comments. For example, bullet comments like "I want this too, how much does the streamer sell it for?" (which don't contain trigger words) can be identified as purchase intentions. For instructions requiring a unique AR event, a time-window-based weighted scoring mechanism determines which user's instruction to execute. The scoring formula is S = (w1·Urank + w2·Gvalue + w3·Chist)·Tdecay. Wherein, S represents the final score, Urank represents the user's level or fan badge level, with a weight w1 = 0.4, Gvalue represents the gift or consumption value associated with the comment, with a weight w2 = 0.5, Chist represents the user's historical interaction frequency, with a weight w3 = 0.1, and Tdecay represents the time decay factor, such as e-0.1t, where t is the number of seconds the instruction waits in the queue to ensure that new instructions have higher priority. In each arbitration cycle, such as 500ms, the instruction with the highest score is adopted, and an AR event trigger signal carrying a unique eventId is generated.

[0169] As the director of the AR interactive experience, a state machine is maintained to track the state of all AR events in each live stream. Upon receiving a trigger signal, the corresponding 3D models, materials, animations, sound effects, and other resources are loaded from the resource database. For example, in the "card unpacking" event, the 3D model of the card pack and the animation sequence of unpacking are loaded. Event parameters are initialized, such as the starting price of the auction item, the increment for each bid, and the auction countdown. Initial rendering instructions are generated, such as {"eventId":"evt_abc","action":"CREATE","assetId":"pack_001","position":[x,y,z],"scale":1.2,"sync_timestamp":1663764900000}, where sync_timestamp can be synchronized based on the Network Time Protocol (NTP) to indicate that the AR object should appear on the video frame at the corresponding time point, thus solving the asynchrony problem caused by network latency.

[0170] The system receives lightweight interactive signaling from a large number of clients in real time. A signaling aggregator collects all interactive signaling related to the same eventId within a very short time window, such as 200ms. For example, it receives 1000 "gesture":"THUMBS_UP" and 500 "gesture":"PALM_OPEN". The aggregated data is then analyzed by the group intent analysis unit to calculate the proportion, frequency, and growth trend of various interactions. For example, in the past 200ms, the proportion of "like" gestures for the evt_abc event reached 66.7%, indicating a rapid increase in popularity. The analysis results, rather than the raw signaling, are passed to the AR event management and synchronization unit. The latter updates the state and rendering instructions of the AR event based on the group intent. For example, a rule is defined: when the "like" popularity exceeds a threshold X, a "glowing" effect is triggered on the auction item, and the intensity of the light I is logarithmically related to the number of users who liked it, I = I base +α·log10(N+1), where, symbol I base The base light intensity is represented by α, and the adjustment coefficient is represented by α. This achieves a smooth visual effect driven by multiple participants. New rendering instructions such as {"eventId":"evt_abc","action":"UPDATE","effect":"glow","intensity":1.5,...} are sent to the rendering stage.

[0171] As a computationally intensive process, it typically runs on dedicated server nodes equipped with top-tier GPUs, maintaining rendering queues to continuously receive rendering instructions. Simultaneously, it obtains decoded, uncompressed raw broadcast video frames, such as YUV420p format, from the access and media processing gateway. For each video frame, it performs environment understanding, lighting estimation, rendering, and compositing. At the same time, it employs a load balancing strategy, with the rendering task scheduler dynamically distributing rendering tasks to different nodes in the GPU cluster based on the popularity and rendering complexity of each live broadcast room, ensuring that the load on a single node is not too high.

[0172] The synthesized video frames are acquired, encoded in real time using H.264 or H.265, encapsulated into streaming media formats such as HLS or DASH, and distributed to global viewers via CDN networks.

[0173] Through the collaborative work between the client and server, the system realizes a complete process from user interaction command collection, parsing, and event triggering to AR content rendering, synthesis, and distribution, providing users with an immersive live interactive experience.

[0174] Example 2: Complete Workflow of the "AR Virtual Card Unboxing" Scenario

[0175] In the "AR virtual card unpacking" scenario, the system achieves a fully immersive interactive experience from activity initialization to the final AR effect presentation through multi-stage collaborative operation. The specific workflow is as follows:

[0176] Step S1: Event Initialization

[0177] The host clearly informed viewers in the live stream that an AR virtual card-unboxing event was about to launch, guiding them through the participation process. The operations team completed the initial configuration of the event through the system backend, including setting the 3D model levels of the virtual card packs involved in the event. These levels were divided into three categories: Common, Rare, and Legendary. Different levels of models differed in detail, initial animation, and special effects. The team also defined the trigger conditions for each level of card pack. For example, viewers needed to gift a virtual gift worth 99 yuan called "Time Key" to trigger the AR card-unboxing event for the Rare level card pack. This configuration information was stored in the system resource library, providing a basis for subsequent event triggering.

[0178] Step S2: User Triggered

[0179] During the live stream, user "User A" gifted a virtual "Time Key" to the streamer via the client interface. The client captured this transaction in real time, encapsulated it into a structured message containing the user identifier ("user_A"), gift type ("Time Key"), gift value (99 yuan), and timestamp, and transmitted it to the server over the network.

[0180] Step S3: Instruction Parsing and Arbitration

[0181] After receiving the signaling message sent by user A's client, the server parses it. Because "gifting a Time Key" is a pre-defined, explicitly triggered action associated with specific consumption value, it receives a high priority score in the system's arbitration mechanism. Since the system does not find any other valid instructions with higher priority within the current time window, it decides to adopt this instruction, generating a unique AR event triggering signaling message. The signaling message includes the event type ("UNBOX," i.e., card opening), the triggering user identifier ("user_A"), and the card pack level ("RARE," i.e., rare). This signaling message is then passed to subsequent processing stages.

[0182] Step S4: AR Event Creation and Synchronization

[0183] The server-side, based on the received AR event trigger signaling, creates a card-unboxing event identified as "evt_unbox_001" to uniquely identify this card-unboxing process. Subsequently, it retrieves a 3D model of a virtual card pack matching the "rare" level from the resource library. This model includes the main appearance of the card pack, the initial floating animation sequence, and the special effects combination during card unboxing. Simultaneously, the system generates initial rendering instructions, specifying the initial placement position of the card pack. This involves automatically identifying the tabletop area in front of the streamer as the physical plane through analysis of the streamer's real-time video stream and positioning the card pack model above this plane. The instructions can also include initial rotation animation parameters for the card pack, such as rotating 30 degrees per second, and a synchronization timestamp based on the Network Time Protocol (NTP) to ensure that all users see the card pack appear at the same time.

[0184] Step S5: Server-side rendering and compositing

[0185] After receiving the initial rendering command, the server-side rendering process begins processing the streamer's live video stream. First, using planar detection technology, the physical coordinates of the streamer's desktop area are confirmed again to ensure the virtual card pack's placement matches the real environment. Next, the lighting information in the video stream is analyzed, detecting that the main light source in the live stream is from the left side of the screen, with moderate ambient light intensity and a warm tone. These lighting parameters are applied to the rendering of the virtual card pack, ensuring the card pack's surface lighting and shadow effects match the real environment. When the video stream plays to the frame corresponding to the synchronization timestamp specified in the command, the system renders a rare-level card pack 3D model on the identified desktop plane and initiates the initial hovering animation—the card pack floats approximately 10 centimeters above the desktop, accompanied by slight up-and-down movement. Finally, the rendered AR visual layer is pixel-level alpha blended with the streamer's live video stream, seamlessly integrating the virtual card pack with the real live stream footage to form a hybrid video stream containing AR content.

[0186] Step S6: Distribute across the entire network

[0187] The synthesized video stream is encoded and encapsulated into an HLS streaming media format suitable for network transmission. Through a Content Delivery Network (CDN), this video stream is distributed to all clients watching the live stream. At this point, all viewers' clients receive the video stream and see a floating, rotating virtual card pack appear on their respective screens in the broadcaster's desktop area, with the card pack's position and animation state perfectly synchronized across all viewers.

[0188] Step S7: User Gesture Interaction

[0189] The host guided the viewers in the live stream, "Please turn on your camera in the client, extend your palms to 'gather energy' for User A's card pack, and help open it!" Viewers "User B", "User C"... "User N" and others responded one after another, turning on their front-facing cameras in the client, pointing their palms at the camera, making the "open palm" gesture, and participating in the interaction.

[0190] Step S8: Lightweight signaling upload

[0191] Each viewer's client's local AI engine processes the camera-captured footage in real time, recognizing the "open palm" hand gesture and generating structured, lightweight signaling, such as: {"eventId":"evt_unbox_001","userId":"user_B","gesture":"PALM_OPEN",...}{"eventId":"evt_unbox_001","userId":"user_C","gesture":"PALM_OPEN",...}. The signaling content includes the corresponding card-opening event identifier ("evt_unbox_001"), user identifier (such as "user_B" or "user_C"), gesture type ("PALM_OPEN"), and recognition confidence level (such as 0.92), and these signaling messages are continuously uploaded to the server.

[0192] Step S9: Interaction Aggregation and State Update

[0193] Within a set 200-millisecond time window, the server continuously receives and aggregates all user gesture signals related to the "evt_unbox_001" event. Statistics show that 5000 "PALM_OPEN" (open palm) gesture signals were received within this time window. The system analyzes this data and arrives at the following group intent analysis result: "The current group is participating in a 'gathering energy' interaction, with an interaction heat value of 5000," and feeds this result back to the event processing stage.

[0194] Step S10: Evolution of AR Effects

[0195] The server-side event handling process generates an update rendering command based on preset rules (when the "Gathering Energy" heat value exceeds 3000, the "Energy Convergence" effect of the card pack is triggered). The command requires adding particle lighting effects to the virtual card pack. The brightness of the lighting effect (1.5 times the base brightness) and the particle density (100 particles per frame) are positively correlated with the current heat value (5000), meaning the higher the heat value, the more pronounced the lighting effect. This update rendering command is then sent to the rendering process.

[0196] Step S11: Rendering Update and Final Presentation

[0197] The rendering process executes update rendering commands, rendering particle light effects around the virtual card pack that increase in intensity with the amount of heat generated. On all viewers' client screens, the card pack's glow gradually intensifies as the number of participants increases. When the heat reaches its peak (e.g., 8000), or when the triggerer "User A" makes a "fist-clenching" gesture (which the client recognizes and uploads the corresponding signaling), the server generates the final command, triggering the card pack's "opening" animation: the card pack gradually unfolds amidst explosive light and shadow effects, revealing the virtual cards one by one, with card information (such as card name and attributes) floating above the card pack in 3D text form. Finally, an augmented reality hybrid video stream containing the complete card-opening process is distributed to all clients, allowing all viewers to simultaneously see the card-opening result, completing the AR virtual card-opening interaction.

[0198] To achieve the technical solutions described above, this invention also provides a technical architecture for implementing the above-described technical solutions, such as... Figure 3 As shown in the diagram, this architecture diagram illustrates the interaction flow between the client and server modules in an AR-integrated immersive live streaming interactive system, as detailed below:

[0199] The client can include a local AI engine, a data acquisition and preprocessing module, and a playback and rendering unit. The local AI engine is used for gesture / facial expression recognition. Specifically, it processes video frames captured by the device's camera in real time, recognizing user gestures (such as liking, opening a palm, clenching a fist, etc.) or facial expressions (such as smiling, surprise, etc.), completing preliminary recognition locally to save bandwidth and protect privacy. The data acquisition and preprocessing module acquires high-definition video streams (such as 1080p, 60fps H.265 encoded video) and audio streams (such as AAC encoded audio) from the broadcaster's end. It also collects the recognition results from the local AI engine, encapsulating them into lightweight biometric signaling. This signaling can contain structured data such as user identifier, gesture / facial expression type, timestamp, and recognition confidence level, and sends it to the server. The playback and rendering unit receives the distributed video stream from the server-side streaming media distribution module, decodes and plays it, presenting a live broadcast with AR interactive content. In some scenarios, it also performs local auxiliary rendering based on server-side instructions, such as displaying customized interactive prompts.

[0200] like Figure 3 As shown, the server side includes a command parsing and event triggering module, an AR event management and synchronization module, a user interaction analysis module, a resource database, a server-side AR rendering and compositing module, and a streaming media distribution module. The contents of each module are described below.

[0201] The instruction parsing and event triggering module receives lightweight biometric signaling from the client, as well as other user instructions such as bullet comments and virtual gift giving. After parsing these instructions, if preset triggering conditions are met, such as giving a specific gift, bullet comments containing triggering keywords, or gesture / expression matching rules, an AR event triggering signal is generated and passed to the AR event management and synchronization module.

[0202] The AR event management and synchronization module receives AR event trigger signals and then initiates a resource query request to the resource database. The resource database stores resources required for AR interaction, such as 3D models (e.g., virtual card packs, auction item models), animation sequences, materials, and sound effects, and returns the corresponding resources based on the request. Simultaneously, the AR event management and synchronization module generates synchronized rendering instructions based on the acquired resources. These instructions can include AR object creation parameters, spatial position, scaling, and animation timing, and are sent to the server-side AR rendering and compositing module. At the same time, the event information is transmitted to the user interaction analysis module.

[0203] The user interaction analysis module receives lightweight biometric signaling from the client and event information from the AR event management and synchronization module. It aggregates and analyzes a large amount of user interaction data, such as gestures, expressions, and bullet screen interactions from multiple users, to obtain group intent analysis results, such as "the user's heat value for participating in 'gathering energy' is 5000, triggering the card pack energy gathering effect," and feeds it back to the AR event management and synchronization module to adjust the AR event status and subsequent rendering instructions.

[0204] The server-side AR rendering and compositing module receives synchronized rendering commands and pre-processed broadcast video / audio streams (data from the broadcast stream that has undergone server-side transcoding, screenshotting, and other pre-processing). First, it performs environmental understanding and lighting estimation on the broadcast video stream. Then, it renders 3D models and executes animation sequences, such as card pack hovering and rotation, according to rendering commands, and handles the depth occlusion relationship between the virtual and real scenes. Finally, it performs pixel-level alpha blending between the AR visual layer and the broadcast video stream to generate a composited augmented reality video stream, which is then passed to the streaming media distribution module.

[0205] The streaming media distribution module is used to receive the synthesized augmented reality video stream, encode it with H.265 and encapsulate it into streaming media formats such as HLS, and then transmit the distributed video stream to the client's playback and rendering unit.

[0206] This technical architecture enables the collaboration of various modules on the client and server sides to complete the entire process from user interaction collection, AR event triggering, resource loading, rendering and compositing to video distribution and playback, thus achieving immersive live interactive streaming with AR integration.

[0207] It should be understood that in the methods described in one or more embodiments of this specification, the order of some steps may be adjusted according to actual needs, or some steps may be omitted.

[0208] Based on the same idea, this specification also provides an apparatus corresponding to the above method, which, when applied to a server, may include:

[0209] The data receiving and access module is used to receive user interaction command data, user biometric signaling, and real-time video stream from at least one client; wherein the user biometric signaling is structured data generated by the client through local AI recognition of the collected raw environmental images.

[0210] The instruction parsing module is used to parse the user interaction instruction data to obtain valid AR interaction instructions.

[0211] The event-triggered arbitration module is used to select a target AR interaction instruction from the valid AR interaction instructions according to a preset arbitration mechanism, and generate a unique AR event triggering signal based on the target AR interaction instruction.

[0212] The AR event management and resource scheduling module is used to respond to the AR event triggering signal, schedule the corresponding 3D virtual resources from the pre-set resource database, and generate a synchronous rendering instruction with a timestamp.

[0213] The initial AR rendering and compositing module is used to perform initial AR rendering and image compositing on the real-time video stream of the broadcaster based on the synchronous rendering instructions, and generate an augmented reality hybrid video stream containing initial AR content.

[0214] The user interaction analysis and group intent recognition module is used to receive and aggregate user biometric signaling from multiple clients for the same AR event, and perform statistical analysis on the user biometric signaling for the same AR event within a set time window to obtain the group intent analysis results.

[0215] The AR event state update and instruction generation module is used to update the state parameters of the AR event based on the group intent analysis results and generate corresponding update rendering instructions.

[0216] The dynamic AR rendering and secondary compositing module is used to perform dynamic AR rendering and secondary compositing on the real-time video stream of the broadcaster and the initial AR content based on the update rendering command, so as to update the augmented reality hybrid video stream.

[0217] The streaming media distribution module is used to distribute updated augmented reality hybrid video streams to various clients via a content delivery network.

[0218] This specification also provides an immersive live streaming interactive device that integrates AR. This device is applied to a client and may include:

[0219] The data acquisition module is used to collect user interaction command data, raw environmental images, and real-time video streams from the broadcaster.

[0220] The local AI recognition and feature generation module is used to recognize the original environmental image through a local AI engine and generate user biometric signaling, wherein the user biometric signaling is structured data.

[0221] The data upload module is used to upload the user interaction command data and the user biometric signaling to the server.

[0222] The hybrid video stream receiving and playback module is used to receive and play the augmented reality hybrid video stream from the server. The augmented reality hybrid video stream is generated by the server based on the user interaction command data and user biometric signaling, after performing AR rendering and image synthesis on the real-time video stream from the broadcaster.

[0223] The augmented reality hybrid video stream includes AR content that is dynamically updated based on the results of group intent analysis. The results of group intent analysis are obtained by the server aggregating and statistically analyzing the biometric signaling of multiple clients for the same AR event.

[0224] It is understood that the modules mentioned above refer to computer programs or program segments used to perform one or more specific functions. Furthermore, the distinction between these modules does not imply that the actual program code must also be separate.

[0225] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will be able to make various obvious changes, readjustments, and substitutions without departing from the scope of protection of the present invention. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. An immersive live streaming interaction method integrating AR, applied on the server side, characterized in that, include: The system receives user interaction command data, user biometric signaling, and real-time video stream from at least one client; wherein the user biometric signaling is lightweight structured data generated by the client through a local lightweight AI engine that identifies the collected raw environmental images. The user interaction command data is parsed to obtain valid AR interaction commands; According to the preset arbitration mechanism, a target AR interaction instruction is selected from the valid AR interaction instructions, and a unique AR event triggering signal is generated based on the target AR interaction instruction. The AR event triggering signal is used to uniquely identify the triggering and execution of a single AR event. In response to the AR event trigger signal, the corresponding 3D virtual resources are scheduled from the pre-set resource database, and a synchronous rendering instruction with a timestamp is generated; Based on the synchronous rendering instructions, initial AR rendering and image synthesis are performed on the real-time video stream of the broadcaster to generate an augmented reality hybrid video stream containing initial AR content; the initial AR rendering includes environmental understanding of the broadcaster's real-time video stream, analysis of lighting information, and depth testing of virtual resources and the real environment; Receive and aggregate user biometric signaling from multiple clients for the same AR event, and perform statistical analysis on the user biometric signaling for the same AR event within a set time window to obtain the group intent analysis results; Update the state parameters of the AR event based on the group intent analysis results, and generate corresponding update rendering instructions; Based on the update rendering instructions, dynamic AR rendering and secondary compositing are performed on the real-time video stream of the broadcaster and the initial AR content to update the augmented reality hybrid video stream; the dynamic AR rendering and secondary compositing include real-time lighting analysis updates, depth occlusion relationship updates between virtual resources and the real environment, and pixel-level alpha blending of the AR visual layer and the broadcaster video stream. The updated augmented reality hybrid video stream is distributed to each client via a content delivery network.

2. The immersive live streaming interactive method integrating AR as described in claim 1, characterized in that, The step of parsing the user interaction command data to obtain valid AR interaction commands includes: The bullet screen text in the received user interaction command data is cleaned and segmented to filter out invalid information and obtain the cleaned bullet screen text. The AC automaton algorithm is used to match preset keywords in the cleaned bullet screen text, and a lightweight text classification model is used to identify the potential instruction intent in the cleaned bullet screen text; the potential instruction intent is an AR event trigger-related intent; Based on the matching results of the preset keywords and the recognition results of the potential instruction intent, a valid AR interaction instruction that meets the preset conditions is obtained.

3. The immersive live streaming interactive method integrating AR as described in claim 2, characterized in that, The step of selecting a target AR interaction command from the valid AR interaction commands according to a preset arbitration mechanism includes: A weighted scoring mechanism based on time windows is adopted to calculate the weighted score for each valid AR interaction command; Within the time window, the valid AR interaction instruction with the highest weighted score is selected as the target AR interaction instruction; The weighted scoring factors include the user level that sent the valid AR interaction command, the consumption value associated with the valid AR interaction command, the user's historical interaction frequency, and the time decay factor.

4. The immersive live streaming interactive method integrating AR as described in claim 1, characterized in that, The process involves receiving and aggregating user biometric signaling from multiple clients for the same AR event, performing statistical analysis on the user biometric signaling for the same AR event within a set time window, and obtaining group intent analysis results, including: Within the set time window, the quantity and type of various user biometric signaling related to the same AR event are counted to obtain statistical results; Based on the statistical results, the proportion, frequency of occurrence, and popularity trend of various interactive behaviors are calculated. The analysis results representing the highest-level group interaction intentions are output as the group intention analysis results.

5. The immersive live streaming interactive method integrating AR as described in claim 1, characterized in that, The process of performing initial AR rendering and image compositing on the real-time video stream of the broadcaster based on the synchronous rendering instructions to generate an augmented reality hybrid video stream containing initial AR content includes: Load the corresponding 3D virtual resources according to the synchronous rendering instructions; The real-time video stream from the broadcaster's terminal is subjected to environmental understanding, and the physical plane of the environment where the broadcaster is located is identified through synchronous positioning and mapping or plane detection. Analyze the lighting information of the real-time video stream, including the direction of the main light source, ambient light intensity, and color; Based on the timestamp and position parameters in the synchronous rendering instruction, the 3D virtual resources are rendered on the physical plane, and depth testing is performed to realize the occlusion relationship between virtual objects and the real environment. The rendered AR visual layer is pixel-level aligned and alpha-blended with the real-time video stream from the broadcaster's end to generate the augmented reality hybrid video stream containing the initial AR content.

6. An immersive live streaming interaction method integrating AR, applied to a client, characterized in that, include: Collect user interaction command data, raw environmental images, and real-time video stream from the broadcaster's end; The original environmental image is identified by a local AI engine to generate user biometric signaling, which is structured data; the user biometric signaling is lightweight structured data. The user interaction command data and the user biometric signaling are uploaded to the server. Receive and play augmented reality hybrid video streams from the server; the augmented reality hybrid video streams are generated by the server based on the user interaction command data and user biometric signaling, after performing AR rendering and image synthesis on the real-time video stream from the broadcaster. The augmented reality hybrid video stream includes AR content that is dynamically updated based on the results of group intent analysis. The results of group intent analysis are obtained by the server aggregating and statistically analyzing the biometric signaling of multiple clients for the same AR event.

7. An immersive live streaming interactive method integrating AR, applied to a network system including a client and a server, characterized in that, include: The client collects user-input interaction command data and raw environmental images obtained through the camera. It uses a local AI engine to recognize the raw environmental images, generates structured user biometric signaling, and uploads the interaction command data and user biometric signaling to the server. At the same time, the client sends a playback request to the server for the live video stream from the broadcaster. The server receives user interaction command data, user biometric signaling, and real-time video streams uploaded by the broadcaster from the client. The system parses user interaction command data, identifies valid AR interaction commands, selects target AR interaction commands from them based on a preset arbitration mechanism, and generates unique AR event triggering signals. In response to the AR event trigger signal, the server schedules the corresponding 3D virtual resources from the resource database, generates a timestamped synchronous rendering instruction, and performs initial AR rendering and image synthesis on the real-time video stream of the broadcaster based on the instruction, generating an augmented reality hybrid video stream containing initial AR content; the initial AR rendering includes environmental understanding, lighting analysis, and depth testing. The server receives and aggregates user biometric signaling from multiple clients for the same AR event, performs statistical analysis within a set time window to obtain group intent analysis results, updates the state parameters of the AR event based on these results, generates update rendering instructions, and performs dynamic AR rendering and secondary compositing on the real-time video stream from the broadcaster's end and the rendered initial AR content accordingly, updating the augmented reality hybrid video stream. The secondary compositing includes pixel-level alpha blending. Finally, the updated video stream is distributed to each client through a content delivery network. The client receives and decodes the augmented reality hybrid video stream from the server; in scenarios requiring personalized interaction, it receives independent AR rendering instructions from the server and overlays personalized interactive elements visible only to the current user onto the locally played hybrid video stream.

8. An immersive live streaming interactive device integrating AR, applied on the server side, characterized in that, include: The data receiving and access module is used to receive user interaction command data, user biometric signaling, and real-time video stream from at least one client; wherein, the user biometric signaling is lightweight structured data generated by the client through the recognition of the collected raw environmental images by the local lightweight AI engine. The instruction parsing module is used to parse the user interaction instruction data to obtain valid AR interaction instructions; The event triggering arbitration module is used to select a target AR interaction instruction from the valid AR interaction instructions according to a preset arbitration mechanism, and generate a unique AR event triggering signaling based on the target AR interaction instruction. The AR event triggering signaling is used to uniquely identify the triggering and execution of a single AR event. The AR event management and resource scheduling module is used to respond to the AR event triggering signal, schedule the corresponding 3D virtual resources from the pre-set resource database, and generate a synchronous rendering instruction with a timestamp. The initial AR rendering and compositing module is used to perform initial AR rendering and image compositing on the real-time video stream of the broadcaster based on the synchronous rendering instructions, and generate an augmented reality hybrid video stream containing initial AR content; the initial AR rendering includes environmental understanding of the broadcaster's real-time video stream, lighting information analysis, and depth testing of virtual resources and the real environment; The user interaction analysis and group intent recognition module is used to receive and aggregate user biometric signaling from multiple clients for the same AR event, and perform statistical analysis on the user biometric signaling for the same AR event within a set time window to obtain the group intent analysis results. The AR event status update and instruction generation module is used to update the status parameters of the AR event based on the group intent analysis results and generate corresponding update rendering instructions. The dynamic AR rendering and secondary compositing module is used to perform dynamic AR rendering and secondary compositing on the real-time video stream of the broadcaster and the initial AR content based on the update rendering command, so as to update the augmented reality hybrid video stream; the dynamic AR rendering and secondary compositing includes real-time lighting analysis update, depth occlusion relationship update between virtual resources and real environment, and pixel-level alpha blending of AR visual layer and broadcaster video stream. The streaming media distribution module is used to distribute updated augmented reality hybrid video streams to various clients via a content delivery network.

9. An immersive live streaming interactive device integrating AR, applied to a client, characterized in that, include: The data acquisition module is used to collect user interaction command data, raw environmental images, and real-time video streams from the broadcaster's end. The local AI recognition and feature generation module is used to recognize the original environmental image through a local AI engine and generate user biometric signaling, wherein the user biometric signaling is lightweight structured data; The data upload module is used to upload the user interaction command data and the user biometric signaling to the server. The hybrid video stream receiving and playback module is used to receive and play the augmented reality hybrid video stream from the server. The augmented reality hybrid video stream is generated by the server based on the user interaction command data and user biometric signaling, after performing AR rendering and image synthesis on the real-time video stream from the broadcaster. The augmented reality hybrid video stream includes AR content that is dynamically updated based on the results of group intent analysis. The results of group intent analysis are obtained by the server aggregating and statistically analyzing the biometric signaling of multiple clients for the same AR event.

Images