Multi-person live chorus method and device, electronic equipment and medium
By dynamically determining user roles and selecting appropriate data synchronization channels within the live streaming platform, the problem of resource waste in ultra-large-scale live streaming is solved, achieving the best balance between cost and quality and supporting large-scale live streaming events.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-05
- Publication Date
- 2026-04-07
AI Technical Summary
Existing technologies cannot support ultra-large-scale live events, and low-latency RTC links result in a significant waste of network and computing resources.
By responding to the commands of the chorus interaction control, the user's interactive participation role on the live streaming platform is dynamically determined, and the target data synchronization channel is selected based on the role. Auxiliary enhancement information and audio and video streams are transmitted in the same bitstream to achieve on-demand allocation of resources, and data synchronization is performed using RTC or CDN channels.
It effectively reduces the number of connections, achieving the best balance between cost, quality, and experience, supporting large-scale live streaming events, and reducing the waste of network and computing resources.
Smart Images
Figure CN121814976A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of network live broadcast, in particular to a method and device for multi-person live broadcast singing, an electronic device and a medium. BACKGROUND
[0002] With the progress of network communication technology, network live broadcast has become a new online interactive way. In live broadcast, in order to realize the extreme synchronization experience (including audio and video, lyrics progress, scores, etc.) between the lead singer, chorus singers and all the audience, the live broadcast system establishes an RTC (Real-Time Communication) connection for all users (including lead singers, chorus singers and audience) in the live broadcast room, and realizes synchronization of data messages.
[0003] Under this architecture, the entire live broadcast room is a fully interconnected and low-latency communication network. When the number of audience is huge (for example, up to tens of thousands or even hundreds of thousands), the bandwidth and resource costs generated by maintaining a large number of RTC connections will increase exponentially, bringing huge economic pressure to the operator. There is an upper limit to the number of connections in a single RTC room, which limits the maximum audience capacity of the live broadcast room and cannot support super large-scale (such as more than 100,000) live broadcast activities. For the audience who do not need real-time interaction, they do not need to upload audio and do not need to receive low-latency dry sound stream (which refers to the original voice signal without any effect processing), and establish a low-latency RTC connection, which causes a great waste of network and computing resources. SUMMARY
[0004] Therefore, the present application provides a method and device for multi-person live broadcast singing, an electronic device and a medium to solve the problem that related technologies cannot support super large-scale live broadcast activities and low-latency RTC links cause a great waste of network and computing resources.
[0005] In a first aspect, the present application provides a method for multi-person live broadcast singing, which comprises: determining the current interactive participation role of the user on the live broadcast platform in response to the opening instruction of the chorus interactive control; determining the target data synchronization channel based on the interactive participation role, wherein the target data synchronization channel is used to synchronize the audio and video streams transmitted in the live broadcast process and the auxiliary enhancement information of the audio and video streams, and the auxiliary enhancement information and the audio and video streams are embedded in the same code stream; obtaining the synchronization data of the live broadcast singing based on the auxiliary enhancement information and the audio and video streams in response to the data transmission of the target data synchronization channel.
[0006] Further, the interactive participation role includes a first interactive participation role and a second interactive participation role. In response to the opening instruction of the chorus interaction control, the current interactive participation role of the user in the live broadcast platform is determined, including: In response to the triggering operation of the first chorus interaction control in the chorus display interface, it is determined that the user currently belongs to the first interactive participation role in the live broadcast platform. In response to the triggering operation of the second chorus interaction control in the chorus display interface, it is determined that the user currently belongs to the second interactive participation role in the live broadcast platform.
[0007] Further, the target data synchronization channel includes a first data synchronization channel and a second data synchronization channel. Based on the interactive participation role, the target data synchronization channel is determined, including: When the interactive participation role is the first interactive participation role, the channel type of the target data synchronization channel is determined to be the first data synchronization channel, wherein the first data synchronization channel is used to synchronize the audio and video streams and the auxiliary enhancement information of the audio and video streams between the user and other chorus participants in the live broadcast platform. When the interactive participation role is the second interactive participation role, the channel type of the target data synchronization channel is determined to be the second data synchronization channel, wherein the second data synchronization channel is used to synchronize the audio and video streams and the auxiliary enhancement information of the audio and video streams in the network where the second data synchronization channel is located to the user in the case of receiving the user's live broadcast request.
[0008] Further, in response to the data transmission of the target data synchronization channel, the synchronization data of the live chorus is obtained based on the auxiliary enhancement information and the audio and video stream, including: Based on the channel type of the target data synchronization channel, the key information to be synchronized is determined; Based on the key information, the auxiliary enhancement information is obtained; Based on the auxiliary enhancement information and the audio and video stream, the synchronization data is determined.
[0009] Further, based on the auxiliary enhancement information and the audio and video stream, the synchronization data is determined, including: Decode the audio and video stream transmitted by the first data synchronization channel to obtain the first audio and video information to be synchronized; The auxiliary enhancement information transmitted by the first data synchronization channel is parsed to obtain the first key information to be synchronized; Synchronize the first audio and video information and the first key information between the user and other chorus participants in the live broadcast platform.
[0010] Further, based on the auxiliary enhancement information and the audio and video stream, the synchronization data is determined, including: Decode the audio and video stream transmitted by the second data synchronization channel to obtain the second audio and video information to be synchronized; The auxiliary enhancement information transmitted through the second data synchronization channel is analyzed to obtain the second key information to be synchronized; Synchronize second audio and video information and second key information with the user.
[0011] Furthermore, the auxiliary enhancement information transmitted through the second data synchronization channel is analyzed to obtain the second key information to be synchronized, including: The auxiliary enhancement information transmitted through the second data synchronization channel is parsed to obtain the service domain identifier; Compare the business domain identifier with the preset identifier of the corresponding business scenario of the live streaming platform; If the business domain identifier matches the preset identifier, the second key information is obtained.
[0012] Secondly, this application provides a device for multi-person live chorus singing, the device comprising: The first determination module is used to determine the user's current interactive participation role on the live streaming platform in response to the activation command of the chorus interaction control; The second determining module is used to determine the target data synchronization channel based on the interactive participant role. The target data synchronization channel is used to synchronize the audio and video streams transmitted during the live broadcast as well as the auxiliary enhancement information of the audio and video streams. The auxiliary enhancement information and the audio and video streams are embedded in the same bitstream. The synchronization module is used to respond to the data transmission of the target data synchronization channel and obtain the synchronization data of the live chorus based on auxiliary enhancement information and audio and video streams.
[0013] Thirdly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the method of multi-person live chorus singing described in the first aspect or any corresponding embodiment.
[0014] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to perform the method of multi-person live chorus as described in the first aspect or any corresponding embodiment.
[0015] Fifthly, this application provides a computer program product, including computer instructions for causing a computer to execute the method of multi-person live chorus singing described in the first aspect or any corresponding embodiment.
[0016] In this embodiment, the user's current interactive role on the live streaming platform is determined in response to the activation command of the chorus interaction control. Then, a target data synchronization channel is determined based on the interactive role. This allows for the transmission of audio and video streams, along with auxiliary enhancement information, through the determined target data synchronization channel. In response to this transmission process, synchronized data for the live chorus is obtained based on the auxiliary enhancement information and the audio and video streams. Therefore, this embodiment dynamically adopts the optimal data synchronization channel based on the user's role in the live stream, achieving "on-demand allocation" of resources. Adaptive data synchronization channels are used according to different real-time requirements of the interactive scenarios, ensuring that different types of network and computing resources are used most appropriately, effectively reducing the number of connections, and thus achieving the best balance between cost, quality, and experience. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this application, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0018] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application; Figure 2 This is a schematic flowchart of a method for live streaming chorus performances according to an embodiment of this application; Figure 3 This is a diagram of the user client's chorus interaction control display interface according to an embodiment of this application; Figure 4 This is a schematic flowchart of another method for live streaming chorus according to an embodiment of this application; Figure 5 This is a structural block diagram of a multi-person live chorus device according to an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0020] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.
[0021] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.
[0022] In a real-time choral live streaming solution based on pure RTC connections, the audio of the lead singer and chorus members is mixed on the RTC server. The mixed audio stream, along with various synchronization data messages (sent via data channels), is then distributed to each viewer. In this architecture, the entire live streaming room is a fully interconnected, low-latency communication network.
[0023] This type of fully interconnected, low-latency communication network will bring the following problems: High cost: Billing for RTC services is typically related to the number of connections and traffic. When the number of viewers is huge (e.g., reaching thousands or even tens of thousands), the bandwidth and resource costs of maintaining massive RTC connections will increase exponentially, putting enormous financial pressure on operators.
[0024] Poor scalability: Since a live streaming room is a single room space, there is an upper limit to the number of connections within that room space. This limits the maximum audience capacity of the live streaming room and makes it unable to support ultra-large-scale (such as more than 100,000 people) live streaming events.
[0025] Waste of resources: For the vast majority of viewers who only want to watch and listen without needing real-time interaction, establishing low-latency RTC connections is unnecessary. They do not need to upload audio or receive low-latency dry audio streams, resulting in a significant waste of network and computing resources.
[0026] Based on this, this application proposes a method for multi-person live chorus as an optional application scenario, such as... Figure 1 As shown, the application scenario includes the broadcaster client 101, server 102 and user client 103 provided in this application embodiment. The broadcaster client 101 and user client 103 interact through the server 102.
[0027] It should be noted that there are multiple interpretations of the concept of "client" in the existing technology. For example, it can be understood as an application installed on a computer device, or it can be understood as a hardware device corresponding to a server.
[0028] In the embodiments of this application, the term "client" refers to a hardware device corresponding to a server, and more specifically, to a computer device, such as a smartphone, a smart interactive whiteboard, and a personal computer.
[0029] When the client is a mobile device such as a smartphone or smart interactive whiteboard, users can install a matching mobile application on the client or access a web application on the client.
[0030] When the client is a non-mobile device such as a personal computer (PC), the user can install the matching PC application on the client, and can also access the web application on the client.
[0031] Among them, mobile applications refer to applications that can be installed on mobile devices, PC applications refer to applications that can be installed on non-mobile devices, and web applications refer to applications that need to be accessed through a browser.
[0032] Specifically, web applications can be divided into mobile and PC versions depending on the client type, and the page layout and server support provided may differ between the two.
[0033] In this embodiment, the live streaming applications provided to users are categorized into mobile live streaming applications, PC live streaming applications, and web live streaming applications. Users can choose their preferred method of participating in live streaming based on the type of client they use.
[0034] Based on the different user identities used by the client, this application can divide the client into broadcast client 101 and user client 103.
[0035] In this context, the broadcaster client 101 refers to the end that sends the live video stream, typically the client used by the broadcaster (i.e., the live streamer user) in a live stream; the user client 103 refers to the end that receives and watches the live video stream, typically the client used by the group watching the video in a live stream. Here, in this embodiment, all other people in the live stream besides the broadcaster are referred to as "users," so the user client 103 can be the client of other participants in the chorus or the client of ordinary viewers.
[0036] Server 102, acting as a business server, can further connect to related audio data servers, video streaming servers, and other servers providing related support, thus forming a logically interconnected service cluster to serve related terminal devices, such as… Figure 1 The broadcaster client 101 and user client 103 shown provide services.
[0037] In this embodiment, the broadcaster client 101 and the user client 103 can join the same live broadcast room (i.e., live broadcast channel). The aforementioned live broadcast room refers to a chat room implemented using Internet technology, which typically has audio and video playback control functions. The broadcaster conducts live broadcasts in the live broadcast room through the broadcaster client 101, and users of the user client 103 can log in to the server 102 to enter the live broadcast room and watch the live broadcast.
[0038] Users can click to access the live streaming application installed on the user client 103 and select to enter any live streaming room, triggering the user client 103 to load the live streaming room interface. The live streaming room interface displays interactive controls for singing, which can realize the switching of interactive participants on the live streaming platform.
[0039] Server 102 responds to the user's trigger operation on the chorus interaction control, determines the target data synchronization channel based on the current user's interactive participation role, then responds to the data transmission of the target data synchronization channel, and synchronizes the data to be synchronized to the user client 103 for display based on the target data synchronization channel.
[0040] According to an embodiment of this application, a method for live multi-person chorus is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0041] This embodiment provides a method for live streaming multi-person choral singing. Figure 2 This is a flowchart illustrating a method for multi-person live chorus singing according to an embodiment of this application, as shown below. Figure 2 As shown, the process includes the following steps: Step S201: In response to the activation command of the chorus interaction control, determine the user's current interactive participation role on the live streaming platform.
[0042] Optionally, this application embodiment relates to the application scenario of live streaming. The lead singer creates a virtual live streaming room on the live streaming platform and uses the RTC data synchronization channel to realize data transmission and synchronization between the lead singer and other chorus members in the live streaming room.
[0043] Specifically, each user's terminal interface upon entering the live stream will display something like this. Figure 3 The interactive chorus control shown (this control can be a button) determines the user's current interactive role on the live streaming platform based on the selected control type after the user clicks the interactive control. For example, Figure 3 In the process, the user's terminal interface displays two chorus interaction controls: the first chorus interaction control is the "Up" control, and the second chorus interaction control is the "Down" control. After the user clicks the "Up" control, the system responds to the trigger operation of the "Up" control, and the user client 103 sends an up request to the server. The server returns a joining credential for the RTC data synchronization channel. At this time, by calling the interface provided by the RTCSDK and calling the SDK's join method, the credential information is passed in, and the SDK will establish a connection with the RTC server, completing the process of joining the RTC data synchronization channel.
[0044] If a user successfully joins the RTC data synchronization channel, their current interactive role changes to the first interactive role, i.e., the chorus member. In other words, regardless of whether the user has been a viewer in the live stream for some time or has just joined, clicking the "Go On" button will switch their role to the chorus member.
[0045] After a user clicks the "Leave the microphone" button, the system responds to this trigger, switching the user from a singer to a second interactive participant, i.e., an audience member. At this point, the user client actively leaves the RTC room, disconnecting all RTC data synchronization channel connections. Typically, a user who can click the "Leave the microphone" button should initially be a singer; otherwise, they cannot switch to an audience member and achieve this identity change.
[0046] As described above, this application embodiment designs a dynamic connection switching mechanism. When a user goes on stage, the connection seamlessly switches from CDN streaming to RTC; when they go off stage, the connection switches in the opposite direction. This process is fast and smooth, with low perceived latency for the user, greatly improving the continuity of the interactive experience from watching to participating and back to watching.
[0047] Step S202: Determine the target data synchronization channel based on the interactive participant roles. The target data synchronization channel is used to synchronize the audio and video streams transmitted during the live broadcast, as well as the auxiliary enhancement information of the audio and video streams. The auxiliary enhancement information and the audio and video streams are embedded in the same bitstream.
[0048] Optionally, after determining the user's interactive role, it is possible to determine which target data transmission channel is selected when implementing data transmission.
[0049] For example, if the user is currently a chorus member, then the user needs to sing a song with the lead singer and other chorus members, which requires a low-latency network connection. In this case, the channel type of the target data synchronization channel is determined to be the first data synchronization channel, i.e., the RTC data synchronization channel.
[0050] For example, if the user is currently an audience member, then a low-latency network connection is not required. In this case, the channel type of the target data synchronization channel is determined to be the second data synchronization channel, namely the CDN data synchronization channel. The audio and video streams of the lead singer and other choruses, as well as auxiliary enhancement information of the audio and video streams, are synchronously pushed to the user in the CDN (Content Delivery Network).
[0051] In this way, the embodiments of this application realize "on-demand allocation" of resources. RTC is used for interactive scenarios with high real-time requirements, and CDN is used for distribution scenarios that are sensitive to bandwidth resources. This ensures that different types of network and computing resources are used in the most appropriate places, avoiding the problem of "oversupply of resources" for a large number of viewers in pure RTC solutions.
[0052] It should be noted that auxiliary enhancement information is additional data that can be inserted into the video stream to carry custom information related to video frames, such as lyrics, scores, timing information, etc. In this way, the auxiliary enhancement information and the audio and video streams are embedded in the same stream, and the timestamps of the audio frames between the auxiliary enhancement information and the audio and video streams are naturally aligned, thereby achieving extremely high-precision data synchronization.
[0053] Step S203: Respond to the data transmission of the target data synchronization channel and obtain the synchronization data of the live chorus based on the auxiliary enhancement information and audio and video streams.
[0054] Optionally, after determining the target data synchronization channel, if the first data synchronization channel is selected, then the synchronization of audio and video streams and auxiliary enhancement information of the audio and video streams between the user and other chorus participants in the live streaming platform is achieved based on the first data synchronization channel. If the second data synchronization channel is selected, when the user sends a live streaming request to the CDN data synchronization channel, the already merged audio and video streams are pulled from the CDN edge nodes. While decoding the audio and video, the player on the viewer's end analyzes the auxiliary enhancement information that may be contained in each frame of the video in real time, and synchronizes the decoded audio and video streams and the analyzed auxiliary enhancement information of the audio and video streams to the user.
[0055] In this embodiment, the user's current interactive role on the live streaming platform is determined in response to the activation command of the chorus interaction control. Then, a target data synchronization channel is determined based on the interactive role. This allows for the transmission of audio and video streams, along with auxiliary enhancement information, through the determined target data synchronization channel. In response to this transmission process, synchronized data for the live chorus is obtained based on the auxiliary enhancement information and the audio and video streams. Therefore, this embodiment dynamically adopts the optimal data synchronization channel based on the user's role in the live stream, achieving "on-demand allocation" of resources. Adaptive data synchronization channels are used according to different real-time requirements of the interactive scenarios, ensuring that different types of network and computing resources are used most appropriately, effectively reducing the number of connections, and thus achieving the best balance between cost, quality, and experience.
[0056] This embodiment provides a method for acquiring business data, such as Figure 4 The process includes the following steps: Step S401: In response to the activation command of the chorus interaction control, determine the user's current interactive participation role on the live streaming platform. For details, please refer to [link to relevant documentation]. Figure 2 Step S201 of the illustrated embodiment will not be described again here.
[0057] Step S402: Determine the target data synchronization channel based on the interactive participant roles. This target data synchronization channel is used to synchronize the audio and video streams transmitted during the live broadcast, as well as the auxiliary enhancement information for the audio and video streams. The auxiliary enhancement information and the audio and video streams are embedded in the same bitstream. For details, please refer to [link to details]. Figure 2 Step S202 of the illustrated embodiment will not be described again here.
[0058] Step S403: Respond to the data transmission of the target data synchronization channel and obtain the synchronization data of the live chorus based on the auxiliary enhancement information and audio and video streams.
[0059] Specifically, step S403 includes: Step S4031: Based on the channel type of the target data synchronization channel, determine the key information to be synchronized.
[0060] Optionally, for the target data synchronization channel, different channel types correspond to different key information to be synchronized. This key information includes song ID, playback progress, current sentence score, and total score. For the first data synchronization channel, the key information to be synchronized includes the song ID, performance score, and playback progress in the vocalist's audio stream; for the second data synchronization channel, it includes the song ID, playback progress, timestamp, current server time, current sentence score, and total score.
[0061] Step S4032: Based on the key information, obtain auxiliary enhancement information.
[0062] Optionally, the server performs real-time mixing of the dry vocals of the lead singer and chorus singers within the RTC data synchronization channel, and blends it with the accompaniment to generate a complete choral audio stream (optionally, video is also mixed simultaneously). The lead singer encapsulates the key information obtained above into SEI (Supplemental Enhancement Information) frames, obtaining auxiliary enhancement information. The SEI frames are used to embed additional information not required for decoding into the video stream, achieving precise synchronous transmission of audio, video, and text data. Since the auxiliary enhancement information is part of the video stream, its timestamp is naturally aligned with the audio frame's timestamp, thus achieving extremely high-precision data synchronization.
[0063] Then, through the bypass streaming function, the complete audio and video stream after mixing, along with the embedded auxiliary enhancement information, is pushed all the way to the traditional CDN network.
[0064] Viewers will then watch the live stream via CDN streaming. The user's client player decodes the audio and video while simultaneously analyzing any auxiliary enhancement information that may be present in each frame of the video, thus achieving precise data synchronization between the viewer and the performer.
[0065] Step S4033: Determine synchronization data based on auxiliary enhancement information and audio / video streams.
[0066] Optionally, the currently obtained auxiliary enhancement information and audio / video streams are used to determine the synchronization data. It should be noted that, for the first data synchronization channel, since the auxiliary enhancement information it obtains is different from that of the second data synchronization channel, the synchronization data of the first data synchronization channel and the second data synchronization channel will also differ.
[0067] In this embodiment, the SEI frame in the bypass streaming is creatively used to carry synchronization information. Since the SEI information and audio / video data are embedded in the same bitstream, have the same timestamp and transmission path, CDN viewers can achieve highly accurate synchronization of lyrics, progress, scores, and audio / video without additional channels. This solves the persistent problem of data and stream asynchrony in traditional CDN solutions, and the system architecture is simpler and more reliable.
[0068] As an optional embodiment, step S4033 includes: Step a1: Decode the audio and video stream transmitted through the first data synchronization channel to obtain the first audio and video information to be synchronized; Step a2: Analyze the auxiliary enhancement information transmitted through the first data synchronization channel to obtain the first key information to be synchronized; Step a3: Synchronize the first audio and video information and the first key information between the user and other chorus participants on the live streaming platform.
[0069] Optionally, when a viewer clicks the "Go On Microphone" button, the user client initiates a go-on request to the business server and obtains a credential to join the RTC data synchronization channel. Subsequently, the user client first stops pulling the stream from the CDN and joins the RTC data synchronization channel created by the broadcaster through the broadcaster's client 101 (which can also be understood as an RTC room / RTC space using the RTC data synchronization channel for transmission). It then begins establishing low-latency RTC connections with the lead singer and other chorus members for audio and video interaction and data synchronization. At this point, the user's data synchronization source switches from the SEI frames of the CDN stream to data channel messages within the RTC room.
[0070] Specifically, the user, acting as a chorus member, subscribes to the audio and video streams of the lead singer and other chorus members transmitted through the first data synchronization channel. The user then uses the server to decode the audio and video streams to obtain the first audio and video information to be synchronized.
[0071] Since the SEI frame is embedded in the same bitstream as the audio and video streams, the auxiliary enhancement information contained in each video frame can also be obtained. The server can parse the auxiliary enhancement information to obtain information such as the song ID, singing score, and playback progress based on the lead singer's audio stream. This information is then synchronized with the user and other chorus participants on the live streaming platform as the primary key information, facilitating synchronization between the audio and lyrics and enhancing the singing experience.
[0072] On the chorus display interface of the user's and other chorus participants' terminals, the aforementioned first key information can also be displayed synchronously. For example, based on the current playback progress in the parsed SEI frame, the UI display can automatically scroll the lyrics and update the score. This application embodiment adopts a low-cost CDN streaming mode for the vast majority of viewers, establishing RTC connections only between a small number of lead singers and chorus members who need real-time interaction. This effectively reduces the number of connections from "tens of thousands" to "tens of thousands," thereby greatly reducing the expensive RTC bandwidth and resource consumption, making the commercial operation of large-scale real-time chorus live streaming possible.
[0073] As an optional embodiment, step S4033 includes: Step b1: Decode the audio and video streams transmitted through the second data synchronization channel to obtain the second audio and video information to be synchronized; Step b2: Analyze the auxiliary enhancement information transmitted in the second data synchronization channel to obtain the second key information to be synchronized; Step b3: Synchronize the second audio and video information and the second key information with the user.
[0074] Optionally, when a singer clicks the "Leave Microphone" button, the client actively leaves the RTC room, disconnects all RTC connections, and simultaneously re-initiates a streaming request to the CDN network, switching back to CDN viewing mode. At this time, its data synchronization source switches back to SEI frame parsing.
[0075] Specifically, the user, acting as a viewer, needs to obtain the audio and video streams of the lead singer and chorus singing in real time. At this point, the already merged audio and video streams are pulled from the CDN edge node, and the user's client player is used to decode the pulled audio and video streams to obtain the second audio and video information to be synchronized.
[0076] Since the SEI frame is embedded in the same bitstream as the audio and video stream, the auxiliary enhancement information contained in each video frame can also be obtained. The server can parse the auxiliary enhancement information to obtain information such as song ID, playback progress, current sentence score, and total score. This information is then used as secondary key information and secondary audio and video information and synchronized with the user and viewers with the same role as the user.
[0077] The second key information mentioned above can also be displayed synchronously on the chorus display interface of the user's and other viewers' terminals. For example, based on the current playback progress in the parsed SEI frame, the UI display can automatically scroll the lyrics and update the score.
[0078] This application embodiment relies on the high scalability of CDN technology itself. The viewing mode on the audience end is not limited by the number of people in the RTC room, and can easily support millions or even tens of millions of users to watch online at the same time, breaking through the technical bottleneck of pure RTC solution.
[0079] As an optional embodiment, step b2 includes: The auxiliary enhancement information transmitted through the second data synchronization channel is parsed to obtain the service domain identifier; Compare the business domain identifier with the preset identifier of the corresponding business scenario of the live streaming platform; If the business domain identifier matches the preset identifier, the second key information is obtained.
[0080] Optionally, the SEI frame transmitted on the second data synchronization channel may include the following information: [4 bytes] Business Domain ID [4 bytes] Timestamp (milliseconds) [4 bytes] Song ID [4 bytes] Playback progress (milliseconds) [4 bytes] Current server time [2 bytes] Score for the current sentence [2 bytes] Total Score Therefore, the business layer is called back in real time when the SEI frame is parsed. The business layer parses the SEI frame into an analyzable model. Then, it checks whether the business domain identifier (such as business domain ID) in the model is consistent with the preset identifier of the business scenario corresponding to the live streaming platform. For example, if the business scenario is a singing scenario, the corresponding preset identifier is 01. This is to check whether the business domain identifier parsed from the received SEI frame is 01. Only if the business domain identifier is consistent with the preset identifier will the current song ID, current progress, and other information be further parsed on this basis. Otherwise, the received SEI frame is considered to be incorrect and will not be parsed.
[0081] This embodiment also provides a device for multi-person live chorus singing, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0082] This embodiment provides a device for live streaming multi-person choral singing, such as... Figure 5 As shown, it includes: The first determining module 501 is used to determine the user's current interactive participation role on the live streaming platform in response to the opening command of the chorus interactive control. The second determining module 502 is used to determine the target data synchronization channel based on the interactive participant role. The target data synchronization channel is used to synchronize the audio and video streams transmitted during the live broadcast and the auxiliary enhancement information of the audio and video streams. The auxiliary enhancement information and the audio and video streams are embedded in the same bitstream. Synchronization module 503 is used to respond to the data transmission of the target data synchronization channel and obtain the synchronization data of the live chorus based on auxiliary enhancement information and audio and video streams.
[0083] In this embodiment, the user's current interactive role on the live streaming platform is determined in response to the activation command of the chorus interaction control. Then, a target data synchronization channel is determined based on the interactive role. This allows for the transmission of audio and video streams, along with auxiliary enhancement information, through the determined target data synchronization channel. In response to this transmission process, synchronized data for the live chorus is obtained based on the auxiliary enhancement information and the audio and video streams. Therefore, this embodiment dynamically adopts the optimal data synchronization channel based on the user's role in the live stream, achieving "on-demand allocation" of resources. Adaptive data synchronization channels are used according to different real-time requirements of the interactive scenarios, ensuring that different types of network and computing resources are used most appropriately, effectively reducing the number of connections, and thus achieving the best balance between cost, quality, and experience.
[0084] In some optional implementations, the interactive participation role includes a first interactive participation role and a second interactive participation role; the first determining module 501 is used to determine the user's current interactive participation role on the live streaming platform in response to the opening command of the chorus interactive control, including: in response to the triggering operation of the first chorus interactive control in the chorus display interface, determining that the user currently belongs to the first interactive participation role on the live streaming platform; in response to the triggering operation of the second chorus interactive control in the chorus display interface, determining that the user currently belongs to the second interactive participation role on the live streaming platform.
[0085] In some optional implementations, the target data synchronization channel includes a first data synchronization channel and a second data synchronization channel; the second determining module 502 is used to determine the channel type of the target data synchronization channel as the first data synchronization channel when the interactive participant role is the first interactive participant role, wherein the first data synchronization channel is used to synchronize the audio and video streams and auxiliary enhancement information of the audio and video streams between the user and other chorus participants in the live streaming platform; when the interactive participant role is the second interactive participant role, the target data synchronization channel is determined to be the second data synchronization channel, wherein the second data synchronization channel is used to synchronize the audio and video streams and auxiliary enhancement information of the audio and video streams in the network where the second data synchronization channel is located to the user when a live streaming viewing request is received from the user.
[0086] In some optional implementations, the synchronization module 503 is used to determine the key information to be synchronized based on the channel type of the target data synchronization channel; obtain auxiliary enhancement information based on the key information; and determine the synchronization data based on the auxiliary enhancement information and the audio and video streams.
[0087] In some optional implementations, the synchronization module 503 is used to decode the audio and video stream transmitted through the first data synchronization channel to obtain the first audio and video information to be synchronized; to parse the auxiliary enhancement information transmitted through the first data synchronization channel to obtain the first key information to be synchronized; and to synchronize the first audio and video information and the first key information between the user and other chorus participants in the live streaming platform.
[0088] In some optional implementations, the synchronization module 503 is used to decode the audio and video stream transmitted through the second data synchronization channel to obtain the second audio and video information to be synchronized; to parse the auxiliary enhancement information transmitted through the second data synchronization channel to obtain the second key information to be synchronized; and to synchronize the second audio and video information and the second key information to the user.
[0089] In some optional implementations, the synchronization module 503 is used to parse the auxiliary enhancement information transmitted through the second data synchronization channel to obtain the service domain identifier; compare the service domain identifier with the preset identifier of the service scenario corresponding to the live streaming platform; and obtain the second key information if the service domain identifier matches the preset identifier.
[0090] The apparatus for multi-person live chorus provided in this disclosure can execute the method for multi-person live chorus provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.
[0091] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.
[0092] The following is a detailed reference. Figure 6 This diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present disclosure. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 601, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 602 or a program loaded from memory 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of the electronic device. The processor 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0093] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.
[0094] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a memory 608, or installed from a ROM 602. When the computer program is executed by the processor 601, it performs the functions defined in the method for multi-person live chorus singing according to embodiments of this disclosure.
[0095] Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0096] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the method of multi-person live chorus shown in the above embodiments is implemented.
[0097] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.
[0098] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.
Claims
1. A method for live streaming multi-person choral singing, characterized in that, The method includes: In response to the command to enable the chorus interaction control, determine the user's current interactive participation role on the live streaming platform; The target data synchronization channel is determined based on the interactive participants. The target data synchronization channel is used to synchronize the audio and video streams transmitted during the live broadcast, as well as the auxiliary enhancement information of the audio and video streams. The auxiliary enhancement information and the audio and video streams are embedded in the same bitstream. In response to the data transmission of the target data synchronization channel, the synchronization data of the live chorus is obtained based on the auxiliary enhancement information and the audio and video stream.
2. The method according to claim 1, characterized in that, The interactive participation roles include a first interactive participation role and a second interactive participation role; The process of responding to the activation command of the chorus interaction control and determining the user's current interactive participation role on the live streaming platform includes: In response to the trigger operation of the first chorus interaction control in the chorus display interface, determine that the user is currently in the first interactive participation role in the live streaming platform; In response to the triggering operation of the second chorus interaction control in the chorus display interface, it is determined that the user is currently in the second interactive participation role in the live streaming platform.
3. The method according to claim 2, characterized in that, The target data synchronization channel includes a first data synchronization channel and a second data synchronization channel; The determination of the target data synchronization channel based on the interactive participant roles includes: When the interactive participant role is the first interactive participant role, the channel type of the target data synchronization channel is determined to be the first data synchronization channel, wherein the first data synchronization channel is used to synchronize the audio and video streams and auxiliary enhancement information of the audio and video streams between the user and other chorus participants in the live streaming platform. When the interactive participant role is the second interactive participant role, the channel type of the target data synchronization channel is determined to be the second data synchronization channel. The second data synchronization channel is used to synchronize the audio and video streams and auxiliary enhancement information of the audio and video streams in the network where the second data synchronization channel is located to the user when a live streaming viewing request is received from the user.
4. The method according to claim 3, characterized in that, The response to the data transmission of the target data synchronization channel, based on the auxiliary enhancement information and the audio and video streams, obtains the synchronization data for the live chorus, including: Based on the channel type of the target data synchronization channel, determine the key information to be synchronized. Based on the key information, the auxiliary enhancement information is obtained; The synchronization data is determined based on the auxiliary enhancement information and the audio / video stream.
5. The method according to claim 4, characterized in that, Determining the synchronization data based on the auxiliary enhancement information and the audio / video stream includes: The audio and video streams transmitted through the first data synchronization channel are decoded to obtain the first audio and video information to be synchronized; The auxiliary enhancement information transmitted through the first data synchronization channel is parsed to obtain the first key information to be synchronized; The first audio and video information and the first key information are synchronized between the user and other chorus participants on the live streaming platform.
6. The method according to claim 4, characterized in that, Determining the synchronization data based on the auxiliary enhancement information and the audio / video stream includes: The audio and video streams transmitted through the second data synchronization channel are decoded to obtain the second audio and video information to be synchronized; The auxiliary enhancement information transmitted through the second data synchronization channel is parsed to obtain the second key information to be synchronized; The second audio and video information and the second key information are synchronized with the user.
7. The method according to claim 6, characterized in that, The step of parsing the auxiliary enhancement information transmitted through the second data synchronization channel to obtain the second key information to be synchronized includes: The auxiliary enhancement information transmitted through the second data synchronization channel is parsed to obtain the service domain identifier; The business domain identifier is compared with the preset identifier of the business scenario corresponding to the live streaming platform; If the business domain identifier matches the preset identifier, the second key information is obtained.
8. A device for multi-person live-streamed choral singing, characterized in that, The device includes: The first determination module is used to determine the user's current interactive participation role on the live streaming platform in response to the activation command of the chorus interaction control; The second determining module is used to determine the target data synchronization channel based on the interactive participant role, wherein the target data synchronization channel is used to synchronize the audio and video streams transmitted during the live broadcast and the auxiliary enhancement information of the audio and video streams, and the auxiliary enhancement information and the audio and video streams are embedded in the same bitstream; The synchronization module is used to respond to the data transmission of the target data synchronization channel and obtain the synchronization data of the live chorus based on the auxiliary enhancement information and the audio and video stream.
9. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the method of live multi-person chorus as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method of live multi-person chorus as described in any one of claims 1 to 7.