Multi-user interaction voice room implementation method

Abstract

The invention discloses a method for realizing a multi-user interaction voice room, and particularly relates to the technical field of audio and video and streaming media distribution, which comprises the following steps: S1, establishing a real-time audio and video communication link for an interaction anchor, S2, mixing multiple anchor streams into a main output stream at a server, the method comprises the following steps: S1, acquiring room key business state information, S3, packaging the room key business state information into a video standard supplemental enhancement information frame and inserting the video standard supplemental enhancement information frame into a main output stream, S4, distributing the stream through a content distribution network, and S5, analyzing the supplemental enhancement information frame by an audience client and synchronously updating an interface. According to the method, the advantages of low delay of real-time audio and video and high concurrency of a content distribution network are fused, the cost is remarkably reduced while the core interactive experience is guaranteed, accurate synchronization of the service state and the audio and video is realized by supplementing and enhancing the information frame, and the problem that the cost, delay and experience cannot be considered at the same time in the existing scheme is solved.

Description

Technical Field

[0001] This invention relates to the field of audio and video and streaming media distribution technology, and more specifically, to a method for implementing a multi-person interactive voice room. Background Technology

[0002] With the rapid development of internet and real-time communication technologies, interactive entertainment services such as online voice chat rooms and live streaming have become important forms of digital social interaction and content consumption. In such business scenarios, users in the room usually exhibit a clear role differentiation: a few interactive hosts with speaking privileges and a large number of ordinary listeners who only listen and watch.

[0003] Currently, the industry mainly relies on two technical architectures to realize such multi-person interactive voice rooms: One approach is the end-to-end real-time audio and video solution. In this solution, all users in the room, including those who are only listening, access the service through a low-latency real-time audio and video service. Its advantage is that it can guarantee extremely low communication latency between interactive hosts and provide a smooth interactive experience. However, the disadvantages of this solution are very prominent: for the vast majority of silent viewers, using a high-cost dedicated real-time audio and video transmission link is extremely uneconomical. At the same time, the scalability and resistance to traffic surges of the real-time audio and video service are relatively limited when faced with a massive influx of users, and the system construction and bandwidth costs are high. The second option is the end-to-end content delivery network (CDN) solution. In this solution, all audio and video content is pushed and pulled through the CDN. Its advantage lies in its ability to fully utilize the mature global distribution network of the CDN, supporting millions or even tens of millions of concurrent users at a relatively low cost, and it has extremely strong scalability. However, its fatal flaw is the high transmission latency, which is usually on the order of several seconds. This cannot meet the millisecond-level latency requirements of highly interactive scenarios such as real-time dialogue, duets, and PK between interactive broadcasters, which seriously restricts the core interactive gameplay and user experience of the voice room.

[0004] Therefore, there is a clear contradiction in the existing technology: high-cost, low-latency real-time audio and video solutions and low-cost, high-latency content delivery network solutions cannot simultaneously meet the dual requirements of low latency for core interaction and low cost for massive distribution, and there are also challenges in state synchronization. Therefore, this invention provides a method for implementing a multi-person interactive voice room. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art, the present invention provides a method for implementing a multi-person interactive voice room to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for implementing a multi-person interactive voice room, comprising the following steps: S1. Establish a connection to create a real-time audio and video communication link for the interactive anchors in the room, enabling low-latency interaction; S2, Mixing and Generation: The server receives the audio and video streams from all the interactive broadcasters, mixes them, and generates a main output stream. S3, State Encapsulation: Encapsulates key business status information within the room into supplementary and enhanced information frames that conform to video coding standards. S4. Distribution: Insert supplementary enhancement information frames into the video frame sequence of the main output stream, and push the main output stream after inserting supplementary enhancement information frames to the content delivery network for distribution; S5. Synchronization: On the client side of the audience, the main output stream is pulled from the content delivery network, the supplementary enhancement information frames are decoded and parsed, and the local user interface is updated according to the parsed business status information to achieve synchronization between business status and audio and video playback.

[0007] Preferably, in step S3, the key business status information includes at least microphone position status information, which is used to indicate the occupancy status of the microphone position, the identity of the occupant, and the audio status.

[0008] Preferably, the key business status information also includes at least one of the following: event information of users entering and leaving the room, gift-giving events and corresponding special effects information, and room attribute change information.

[0009] Preferably, in step S3, the strategy for inserting supplementary enhancement information frames includes periodic insertion and event-triggered insertion, wherein the event-triggered insertion is performed when events such as microphone position changes, users entering or leaving the room, or gift giving occur.

[0010] Preferably, the payload data of the supplementary enhancement information frame includes identification information for verifying the state update order and / or achieving time alignment.

[0011] Preferably, step S2 further includes: dynamically adjusting the screen layout of the main output stream according to the online status of the interactive anchor and the media type, and generating transcoded streams containing at least two different bitrates for viewers with different network conditions to pull.

[0012] Preferably, before step S1, an access step is also included: in response to a user's request to join the room, based on the user's permissions and system status, a decision is made and the user is guided to access the room through a real-time audio / video link or a content delivery network link.

[0013] Preferably, the method further includes a switching step: when a viewer's request to go on the microphone is approved, the access method is switched from streaming through the content delivery network to uploading audio and video through a real-time audio and video link.

[0014] Preferably, the method further includes a degradation step: when an anomaly is detected in the real-time audio and video link of the interactive anchor, the media streaming method is switched to streaming via a content delivery network.

[0015] Preferably, in step S5, if parsing the supplementary enhancement information frame fails, the client obtains the current room's service status information from the service server through an independent signaling channel to update the user interface.

[0016] The technical effects and advantages of this invention are as follows: 1. This invention divides room users into two roles: interactive anchors and viewers. It uses real-time audio and video links and content delivery network links for service, respectively. It combines the low latency advantage of real-time audio and video with the high concurrency and low cost advantage of content delivery network. The core interactive anchors ensure real-time audio and video interaction with millisecond-level latency through real-time audio and video, ensuring a high-quality interactive experience. Meanwhile, the massive number of viewers pull the synthesized stream through the content delivery network, which greatly reduces the overall bandwidth consumption and server load of the system. Thus, while ensuring the core functional experience, it significantly reduces operating costs. 2. This invention encapsulates key business information such as the dynamically changing microphone position status and gift events within the room into supplementary enhancement information frames in standard video encoding and injects them into the main output audio and video streams distributed to all viewers. This ensures that the business status and audio / video content are transmitted together and arrive in the same frame. When the viewer's client decodes and plays the video, it simultaneously parses the supplementary enhancement information frames and immediately updates the local interface. This fundamentally eliminates the problems of audio-visual asynchrony and business status asynchrony caused by buffering and latency jitter at various levels of the content delivery network. It ensures that all users, regardless of their network conditions, perceive that the room status and audio / video content are strictly aligned on the timeline, greatly improving the consistency of the user experience. 3. The method of this invention has a clear architecture, with real-time audio and video and content delivery network each performing their respective functions and being fully decoupled. On the one hand, the system naturally inherits the global distribution and traffic surge resistance capabilities of the content delivery network, and can easily support ultra-large-scale voice room scenarios with millions of users online simultaneously, exhibiting extremely strong scalability. On the other hand, by introducing mechanisms such as intelligent access decision-making, dynamic role switching, and link anomaly degradation, the elasticity and robustness of the service are ensured when facing network fluctuations or local failures. In addition, since supplemental enhancement information is a component of mainstream video coding standards such as H.264 / H.265, this solution is compatible with common player and content delivery network services, requiring no major modifications to existing infrastructure, making the technology easy to implement and highly practical. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the overall steps of the present invention. Detailed Implementation

[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0019] This invention provides a method for implementing a multi-user interactive voice room. The multi-user interactive voice room system mainly includes the following components: Clients: Divided into interactive broadcaster clients and audience clients. The broadcaster client integrates a real-time audio and video (RTC) SDK for capturing, encoding, and pushing audio and video streams and receiving streams from other broadcasters. The audience client integrates a standard streaming media player SDK (such as supporting FLV / HLS) and a supplementary enhanced information frame parsing module for pulling streams, decoding, rendering, and state synchronization. Business Server: Responsible for handling core business logic such as room management, user authentication, microphone control, gifts, bullet comments, and chat, and maintaining the overall state of the room; Real-time audio and video (RTC) media servers typically employ a Selective Forwarding Unit (SFU) architecture, responsible for receiving high-quality, low-latency audio and video streams from all interactive broadcasters. Mixing service: Receives multiple broadcast audio and video streams from the RTC media server and mixes them into a single continuous main output stream. One of the core functions of this service is to encapsulate the room status information sent by the business server into supplementary enhancement information frames and insert them into the video stream. The mixing service can be integrated with the RTC media server or deployed independently. Content Delivery Network: Receives the main output stream pushed by the mixing service and uses its globally distributed edge nodes to efficiently and stably distribute the stream to a massive audience. Intelligent scheduling gateway: As the entry point for user access, it intelligently decides whether users should access the system through the RTC link or the content delivery network link based on user roles, client capabilities, network conditions, and system load.

[0020] The method steps of this embodiment will be described in detail below, in conjunction with the above system architecture: Before implementing step S1 in this embodiment, an access step is also included: When a user initiates a request to join a voice room through a client application, the request first reaches the intelligent scheduling gateway. The gateway queries the business server for the user's permissions in the current room based on the user's identity identifier carried in the request. For example, whether the user is a host designated by the administrator or a regular viewer. At the same time, the gateway will assess the overall load of the current system, such as RTC service capacity, content delivery network bandwidth utilization, and the network status of the user's terminal. If a user is authorized as an interactive broadcaster and the system has sufficient RTC resources, the gateway will guide the user to establish a connection with the RTC media server and enter the broadcaster process. Otherwise, the user will be directed to the content delivery network's streaming entry point to become a viewer.

[0021] This step enables dynamic resource allocation, ensuring that core users have priority access to low-latency resources while maintaining system stability under high traffic volumes.

[0022] S1. Establish connection Authorized users as interactive broadcasters establish signaling and media connections with the RTC media server through their clients, guided by the gateway. All interactive broadcasters form an audio and video data exchange network through the RTC media server, enabling millisecond-level low-latency interactive communication. S2, Mixed Flow Generation The RTC media server forwards all audio and video streams from the interactive broadcasters to the mixing service. The mixing service combines multiple video streams into a single image based on preset rules or dynamic logic (such as a close-up of the current speaker). For example, it may use a grid layout or speaker mode. The audio streams are then mixed into a single continuous audio track. To accommodate viewers with different network conditions, the mixing service or its associated transcoding service will transcode the synthesized main output stream to generate at least two different bitrate streams (e.g., 1080p@3Mbps, 720p@1.5Mbps, 480p@800kbps). These streams have the same supplementary enhancement information frame content, but different video bitrates. Finally, one (or multiple) main output streams containing audio and video at different bitrates are generated, waiting to be distributed; S3, State Encapsulation Status information definition: The business server maintains the global status of the room. When the status changes (for example, user A goes on the microphone, user B sends a gift, user C leaves the room), the business server will generate a status update message. This message is structured data and must contain microphone status information, and may also contain instantaneous event information. The business server sends status update messages to the mixing service in real time. The mixing service uses this message as a payload and encapsulates it into a supplemental enhancement information frame that conforms to the H.264 / H.265 video coding standard. The supplemental enhancement information frame is part of the video stream and is transmitted along with the image data. When encapsulating data, structured formats such as JSON and Protocol Buffers can be used. A simplified JSON example is shown below: json { "type": "room_status", / / Message type "timestamp": 1678886400123, / / Server timestamp "data": { "mic_positions": [ / / Microphone position array {"index": 0, "user_id": "user123", "is_muted": false}, {"index": 1, "user_id": "user456", "is_muted": true}, {"index": 2, "user_id": null, "is_muted": true} ], "gift_effect": {"sender": "user789", "gift_id": "gift_heart"} / / Gift effect } } Add identification information: The load data must include an auto-incrementing sequence number and a server timestamp. The sequence number is used by the client to verify the order and integrity of status updates to prevent out-of-order or packet loss due to network jitter. The timestamp is used for precise time alignment in multi-stream or playback scenarios. The mixing service inserts encapsulated supplementary enhancement information frames into the video stream using a hybrid insertion strategy: Periodic insertion: For example, inserting a supplementary enhancement information frame containing the complete room status once per second as a status heartbeat and fallback synchronization mechanism for the viewer; Event-triggered insertion: Once a status update notification (such as microphone position change, gift event) is received from the business server, a supplementary enhancement information frame containing the incremental or latest full status is immediately generated and inserted, ensuring that key interactive events can be notified to all viewers very quickly (within the next video frame interval).

[0023] S4, Distribution The mixing service inserts supplementary enhancement information frames into the main output stream (and multiple bitrate versions), and pushes it to the origin server of the content delivery network using standard streaming media protocols (such as RTMP, SRT). The content delivery network is responsible for distributing the stream to global edge nodes, and has the ability to handle high concurrency and withstand sudden traffic surges. S5, Synchronization The listening client selects a content distribution network stream with an appropriate bitrate based on its own network conditions, and the standard player decodes and renders the audio and video streams. The integrated supplemental enhancement information frame parsing module detects and extracts supplemental enhancement information payload data in video frames in real time during the video decoding process. The parsing module parses out structured state data (such as JSON), reads the sequence number and timestamp for verification, and then the client application immediately updates the local UI based on this data: for example, changing the avatar of microphone position 1 to user A, playing the super rocket effect gifted by user B on the screen, updating the online user list, etc. If the client fails to parse the supplementary enhancement information frame for an extended period of time, or if parsing fails (e.g., due to incorrect data format), it will automatically degrade: it will actively request the latest status of the current room from the business server through an independent WebSocket or HTTP long-connection signaling channel to ensure the eventual consistency of UI information.

[0024] This invention also includes a switching step: When a viewer clicks the "Request to Go on Microphone" button, and the host or system approves it, the business server notifies the user's client and the smart dispatch gateway. Upon receiving the notification, the client performs parallel operations: on one hand, it maintains the existing content delivery network (CDN) streaming to continue listening to the room content; on the other hand, it launches the RTC SDK and initiates a request to the smart scheduling gateway to establish an RTC connection. After the gateway verifies the connection, it guides the client to establish a connection with the RTC media server. Once the connection is successful, the client begins uploading its own audio and video streams. The business server notifies the mixing service of this change, and the mixing service adds the new streamer's stream to the mixing stream. It also notifies all viewers that the microphone position status has been updated by supplementing enhancement information frames. Thus, the user has seamlessly switched from being a CDN streamer to an RTC uploading streamer.

[0025] This invention also includes a downgrade step: The system continuously monitors the RTC link quality (such as latency and packet loss rate) of each interactive broadcaster. When the system detects that a broadcaster's link quality is consistently below a preset threshold, or when the RTC service fails, it triggers a degradation process to ensure that the broadcaster's speech can be heard by other users. The service server notifies the broadcaster client: "Please switch to content delivery network streaming." The broadcaster client starts the backup encoder and pushes the local audio and video stream directly to a specific stream address on the content delivery network using a standard streaming protocol (such as RTMP). The mixing service pulls this downgraded stream from the content delivery network address, replaces the original RTC stream, and continues to participate in the mixing. At the same time, the broadcaster is notified of the change in their role status via signaling. This step sacrifices the ultra-low latency two-way interaction between the streamer and other streamers, but ensures that the streamer's one-way speech can continue to be transmitted to all viewers, thus improving the overall robustness of the system.

[0026] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for implementing a multi-user interactive voice chat room, characterized in that: Includes the following steps: S1. Establish a connection to create a real-time audio and video communication link for the interactive anchors in the room, enabling low-latency interaction; S2, Mixing and Generation: The server receives the audio and video streams from all the interactive broadcasters, mixes them, and generates a main output stream. S3, State Encapsulation: Encapsulates key business status information within the room into supplementary and enhanced information frames that conform to video coding standards. S4. Distribution: Insert supplementary enhancement information frames into the video frame sequence of the main output stream, and push the main output stream after inserting supplementary enhancement information frames to the content delivery network for distribution; S5. Synchronization: On the client side of the audience, the main output stream is pulled from the content delivery network, the supplementary enhancement information frames are decoded and parsed, and the local user interface is updated according to the parsed business status information to achieve synchronization between business status and audio and video playback.

2. The method for implementing a multi-user interactive voice room according to claim 1, characterized in that: In step S3, the key business status information includes at least microphone position status information, which is used to indicate the occupancy status of the microphone position, the identity of the occupant, and the audio status.

3. The method for implementing a multi-person interactive voice room according to claim 2, characterized in that: Key business status information also includes at least one of the following: event information of users entering and leaving the room, gift-giving events and corresponding special effects information, and room attribute change information.

4. The method for implementing a multi-user interactive voice room according to claim 1, characterized in that: In step S3, the strategy for inserting supplementary enhancement information frames includes periodic insertion and event-triggered insertion. The event-triggered insertion is performed when events such as microphone position changes, users entering or leaving the room, or gift giving occur.

5. The method for implementing a multi-person interactive voice room according to claim 1, characterized in that: The payload data of the supplemental enhancement information frame contains identification information used to verify the state update order and / or achieve time alignment.

6. The method for implementing a multi-person interactive voice room according to claim 1, characterized in that: Step S2 also includes: dynamically adjusting the screen layout of the main output stream according to the online status of the interactive anchor and the media type, and generating transcoded streams containing at least two different bitrates for viewers with different network conditions to pull.

7. The method for implementing a multi-user interactive voice room according to claim 1, characterized in that: Before step S1, there is also an access step: in response to a user's request to join the room, based on the user's permissions and system status, a decision is made and the user is guided to access the room through a real-time audio / video link or a content delivery network link.

8. The method for implementing a multi-person interactive voice room according to claim 7, characterized in that: The method also includes a switching step: when a viewer's request to go on the microphone is approved, the access method is switched from streaming through the content delivery network to uploading audio and video through a real-time audio and video link.

9. The method for implementing a multi-user interactive voice room according to claim 1, characterized in that: The method also includes a degradation step: when an anomaly is detected in the real-time audio and video link of the interactive anchor, its media streaming method is switched to streaming via a content delivery network.

10. The method for implementing a multi-person interactive voice room according to claim 1, characterized in that: In step S5, if parsing the supplementary enhancement information frame fails, the client obtains the current room's service status information from the service server through an independent signaling channel to update the user interface.

Images