Methods, apparatuses, devices, and media for participating in interactive activities

Abstract

Methods, apparatus, devices, and media for participating in interactive activities are provided. In one method, in response to determining that a user participating in the interactive activity is an audience type, first audio data is provided to the user's terminal device. The first audio data includes background music and voice data of the interactive activity, the voice data originating from at least one audio user participating in the interactive activity. In response to receiving a user request from the user to participate in the interactive activity, the provision of the first audio data to the terminal device is stopped. Second audio data, including the voice data of the interactive activity, is provided to the terminal device. By generating two different audio data sets, user participation in the interactive activity can be supported in a simpler and more efficient manner. Furthermore, in this way, it is not necessary to set an upper limit on the number of users participating in the interactive activity, thereby supporting more users to participate in the interactive activity.

Description

Technical Field

[0001] The implementation of this disclosure generally relates to live web streaming, and in particular to methods, apparatus, devices, and computer-readable storage media for participating in interactive activities. Background Technology

[0002] With the development of live streaming technology, various types of activities can now be participated in within live streaming rooms. For interactive activities, users can establish a connection with the live streaming room and participate in interactive activities. However, during the interaction process, a large amount of data is transmitted between the user's terminal device and the server providing the live streaming service, and the interactive activities consume significant server resources. Due to the workload requirements of the server, it is necessary to set an upper limit on the number of participants. Therefore, a more efficient interactive activity management technology solution is needed to enable more users to participate in interactive activities. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for participating in an interactive activity is provided. In this method, in response to determining that a user participating in the interactive activity is an audience type, first audio data is provided to the user's terminal device. The first audio data includes background music and voice data of the interactive activity, the voice data originating from at least one audio user participating in the interactive activity. In response to receiving a user request from the user to participate in the interactive activity, the provision of the first audio data to the terminal device is stopped. Second audio data is provided to the terminal device, the second audio data including the voice data of the interactive activity.

[0004] In a second aspect of this disclosure, an apparatus for participating in an interactive activity is provided. The apparatus includes: a first providing module configured to provide first audio data to a user's terminal device in response to determining that a user participating in the interactive activity is an audience type; the first audio data including background music and voice data of the interactive activity, the voice data originating from at least one audio user participating in the interactive activity; a stopping module configured to stop providing the first audio data to the terminal device in response to receiving a user request from the user participating in the interactive activity; and a second providing module configured to provide second audio data to the terminal device, the second audio data including the voice data of the interactive activity.

[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes: at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to a first aspect of this disclosure when executed by the at least one processor.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to a first aspect of this disclosure.

[0007] In a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to a first aspect of this disclosure.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the implementation of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] In the following detailed description, the above and other features, advantages, and aspects of the various implementations of this disclosure will become more apparent, taken in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A block diagram of an example environment based on some implementations of this disclosure is shown;

[0011] Figure 2 A block diagram illustrating some implementations of this disclosure for participating in interactive activities is shown;

[0012] Figure 3 A block diagram is shown illustrating some implementations of this disclosure for generating second audio data;

[0013] Figure 4 A block diagram is shown illustrating some implementations of this disclosure for aligning background music and second audio data;

[0014] Figure 5 A block diagram of different users in a live streaming room is shown according to some implementations of this disclosure;

[0015] Figure 6 A block diagram illustrating some implementations of this disclosure for participating in interactive activities is shown;

[0016] Figure 7 A flowchart is shown illustrating a method for participating in interactive activities according to some implementations of this disclosure;

[0017] Figure 8 A block diagram of an apparatus for participating in interactive activities according to some implementations of this disclosure is shown; and

[0018] Figure 9 A block diagram of a device capable of implementing various implementations of the present disclosure is shown. Detailed Implementation

[0019] Implementations of this disclosure will now be described in more detail with reference to the accompanying drawings. While some implementations of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the implementations set forth herein. Rather, these implementations are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and implementations of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0020] In the description of the implementation methods disclosed herein, the term "comprising" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the relationships between various data. For example, the aforementioned relationships can be obtained based on various currently known and / or future-developed technical solutions.

[0021] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0022] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0023] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0024] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, for example, via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose whether to "agree" or "disagree" to provide personal information to the electronic device.

[0025] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0026] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of subsequent actions performed in response to such event or condition is not necessarily strongly correlated with the time when the event occurs or the condition is met. For example, in some cases, subsequent actions may be performed immediately upon the occurrence of the event or the fulfillment of the condition; while in others, they may be performed some time after the occurrence of the event or the fulfillment of the condition.

[0027] Example Environment

[0028] Figure 1 A block diagram 100 illustrates an example environment according to some implementations of this disclosure. In this environment, user 110 can send a request to create a live streaming room (e.g., via a live streaming application) via terminal device 120 and utilize server 150 to provide live streaming services, etc. In some scenarios, user 110 is also referred to as the host, streamer, or administrator of the live streaming room. Taking a choral activity as an example of an interactive activity, in a live streaming activity providing a choral activity, user 110 can be referred to as the lead singer or lead singer user. Terminal device 120 can also be referred to as the host end of the live streaming room.

[0029] One or more users 130-1, 130-2, ..., 130-N can watch the live stream and participate in activities such as singing along through their respective terminal devices 140-1, 140-2, ..., 140-N. For ease of discussion, users 130-1, 130-2, ..., 130-N can be collectively referred to as user 130 or individually, and terminal devices 140-1, 140-2, ..., 140-N can be collectively referred to as terminal device 140 or individually. User 130 can be of various types, such as guest users, users participating in the singing along, and audience users who do not participate in the singing along, etc. Guest users can be invited to join the live stream and interact with the host via voice chat.

[0030] It should be understood that, although Figure 1This document only shows a single live streamer user; in some embodiments, multiple live streamers may participate in a single live stream room. In some embodiments, terminal devices 120 and 140 may each have an application capable of providing live streaming services installed, or access to a website capable of providing live streaming services. Users 110 and 130 can operate terminal devices 120 and 140 to access the corresponding application or website. Accordingly, terminal devices 120 and 140 can present a corresponding live streaming interface, which may provide live streaming content, such as audio or video content.

[0031] In some embodiments, terminal devices 120 and 140 can also communicate with server 150 via network 152 to provide live streaming services. Server 150 can provide functions such as management, configuration, and maintenance of applications or websites. Server 150 may include one or more servers, for example, it may include servers for supporting live streaming services, servers for supporting choral activities, and so on.

[0032] Terminal devices 120 and 140 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal devices 120 and 140 may also support any type of user-facing interface (such as "wearable" circuitry).

[0033] Server 150 can be various types of computing systems / servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc. Server 150 can, for example, provide backend services for applications providing live streaming services in terminal devices 120 and 140. It should be understood that the structure and function of various elements in the environment are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Some exemplary embodiments of this disclosure will continue to be described below with reference to the accompanying drawings.

[0034] It can be like Figure 1The environment shown provides a choral activity where users can connect to the server and participate in a live choral session. However, during the chorus, a significant amount of data is transmitted between the user's terminal device and the server providing the live streaming service, consuming substantial server resources. Therefore, due to server workload requirements, a limit needs to be set on the number of participants (e.g., 30 people or another number). When the limit is exceeded, new users applying to join must wait for existing users to leave the chorus before they can enter. Therefore, a more efficient choral activity management solution is desired to enable more users to participate.

[0035] Summary of Participation in Interactive Activities

[0036] To at least partially address the shortcomings of the prior art, a method for participating in interactive activities is proposed according to one implementation of this disclosure. In the context of this disclosure, interactive activities may include at least one of the following: choral activities or recitation activities. For ease of description, the process of participating in an activity will be described below using only a choral activity as an example of an interactive activity.

[0037] In summary, two distinct audio data sets can be generated. The first audio data set may include background music for the choral activity and vocal data from at least one participating user. The second audio data set may consist solely of vocal data. Here, the vocal user's singing voice is audible to other users. Different audio data can be provided to different types of users, thereby supporting greater user participation in the choral activity.

[0038] See Figure 2 A summary of one implementation according to this disclosure is described. Figure 2 A block diagram 200 illustrating some implementations of this disclosure for participating in interactive activities is shown. For example... Figure 2 As shown, the type of user 230 in the live stream can be determined. In response to determining that the user in the live stream is a viewer, as shown by the dashed line, first audio data 210 can be provided to user 230's terminal device 232. Here, the first audio data 210 may include background music (BGM) and vocal data from at least one vocal user participating in the chorus. User 230 may wish to participate in the chorus 240. In this case, in response to receiving a user request 234 from the user to participate in the chorus, the provision of first audio data 210 to the terminal device can be stopped, and second audio data 220 can be provided to the terminal device 232. Here, the second audio data 220 only includes the vocal data of the chorus but does not include the background music.

[0039] According to some implementations of this disclosure, the methods according to some implementations of this disclosure can be executed on any device with processing capabilities. For example, the proposed methods can be executed on one or more server devices. In this case, user 230 can play background music at terminal device 232 and participate in a chorus along with the background music and received voice data. At this time, user 230's voice data can only be heard by himself and will not be transmitted to other users. In this way, the auditory effect of participating in a chorus activity can be simulated locally at the terminal device. For ease of description, such users can be referred to as "virtual chorus users," who do not connect to the interactive service of the live broadcast room (e.g., real-time communication service) but receive audio data through a passive service (e.g., content delivery network service).

[0040] According to some implementations of this disclosure, by generating two different audio data, user participation in choral singing can be supported in a simpler and more efficient way. Furthermore, this method eliminates the need to set a limit on the number of users participating in choral activities, thereby supporting more users to participate in choral singing.

[0041] Detailed process of participating in interactive activities

[0042] Having outlined some implementations of this disclosure, further details regarding participation in choral activities will be described below. According to some implementations of this disclosure, audio data can be provided to different types of users based on different service connections; that is, different types of users' terminal devices can communicate with the server device via different service connections. For example, there can be a first service connection supporting interaction between the audio user and the live broadcast room, and a second service connection supporting only passive data reception between the audience user and the virtual chorus user and the live broadcast room.

[0043] Specifically, a first service connection exists between the user's terminal device and the server providing the interactive activities, and a second service connection exists between the user's terminal device and the server providing the interactive activities. In a live streaming scenario, real-time communication (also known as the first service, abbreviated as RTC) and a content delivery network (also known as the second service, abbreviated as CDN) can be used. Since the user needs to interact frequently with the live stream, the RTC service can be used to support low-latency real-time interaction between the user and the live stream (the data stream corresponding to the RTC service can be referred to as an RTC stream). According to some implementations of this disclosure, guest users can be present in the live stream, and the RTC service can be used to handle the communication between the guest users' terminal devices and the server devices.

[0044] During the live stream, the RTC service can forward the live chat content to the CDN service, allowing viewers who are not participating in the chorus to access the live content from the CDN service (the data stream corresponding to the CDN service can be referred to as a CDN stream). In other words, chorus data is transmitted to the viewers in the live stream via a second service connection. The chorus data can include the voice data of all users participating in the chorus and background music. At this time, all viewers in the live stream can hear the chorus data. By utilizing some implementation methods disclosed herein, limited resources can be prioritized for users participating in the chorus, thus ensuring the real-time processing performance of the live stream.

[0045] In this disclosure, to support greater user participation in choral singing, two CDN data streams can be generated: a first audio stream including background music and vocal data, and a second audio stream containing only vocal data. Since the CDN server only pushes the generated audio data to users and does not receive feedback from users, increasing the number of virtual chorus users participating in the choral activity via CDN does not significantly increase the server workload. In this way, more users can participate in the choral activity while ensuring sufficient server workload.

[0046] According to some implementation methods of this disclosure, an upper limit can be set for the number of audio users to avoid excessive workload caused by collecting a large amount of audio user voice data in real time. Users can submit user requests to participate in the chorus activity, and it can be determined whether the current number of audio users has reached a predetermined threshold. If the number of audio users has not reached the predetermined threshold, the user request can be accepted, and the user can be set as an audio user and participate in the chorus. An RTC connection can be established between the user and the server device, at which time the voice from that user can be heard by other users in the live broadcast room.

[0047] According to some implementations of this disclosure, if it is determined that the number of current audio users has reached a predetermined threshold, the method of this disclosure can be executed, and the user can be set as a "virtual chorus user," and the provision of first audio data to that user can be stopped. Specifically, in the process of stopping the provision of first audio data to the terminal device, in response to receiving a user request from a user to participate in the chorus activity, the number of at least one audio user currently participating in the chorus activity can be determined. In response to determining that the number of users has reached the predetermined threshold, the provision of first audio data to the terminal device is stopped. In this way, even when it is determined that the number of audio users has reached its limit, CDN services can be used to simulate the auditory effect of users participating in the chorus, thereby supporting more users to participate in the chorus.

[0048] According to some implementations of this disclosure, background music for a choral activity can be provided to a terminal device, and the terminal device can be instructed to launch a player to play the background music. Specifically, background music can be sent to the user's terminal device after receiving a request from the user to participate in the choral activity. Alternatively and / or additionally, background music can be sent to the user in advance at the initial stage of the choral activity. In this way, the user's waiting delay can be reduced and the user can participate in the choral activity immediately.

[0049] A local media player can be launched on the user's device to play background music. The player allows users to adjust background music settings, such as raising or lowering the pitch, and setting mixing effects. Alternatively and / or additionally, users can set the volume, pitch, and duration of the second audio data within the player. For example, a user can choose to play the second audio data only during the chorus of a song, or sing throughout the entire song, simulating the auditory effect of other users singing along. This allows users to participate in choral activities more flexibly according to their preferences.

[0050] According to some implementations of this disclosure, user voice data can be acquired using a local acquisition device (e.g., a microphone) on the user's terminal device. Mixing processing can be performed locally on the user's terminal device to synthesize an audio stream that includes background music, the user's voice data, and voice data from other users. In this way, the auditory effect of a user participating in a choral activity can be simulated locally on the terminal device without significantly increasing the server's workload.

[0051] During a chorus, multiple users can sing on their respective client devices, and their voices can be sent to a server. Because the terminal device status and network status of each user differ, the server receives multiple voice data at different times. Using some implementations of this disclosure, the generation of the second audio data requires aligning the different voice data. Specifically, it can receive first voice data from a first user (at least one of the users) and second voice data from a second user (at least one of the users); and align the first and second voice data to generate the second audio data.

[0052] See Figure 3 To describe more details, the Figure 3 A block diagram 300 for generating second audio data is shown, according to some implementations of this disclosure. For example... Figure 3As shown, assume there are two users with audio input, 310 and 320, meaning their voices can be heard by other users. Voice data 312 from user 310 and voice data 322 from user 320 can be collected. Furthermore, voice data 312 and 322 can be aligned to generate second audio data 220. Specifically, the timestamp of the background music can be used as the standard time to generate second audio data that matches the rhythm of the background music. In this way, even though voice data from different users is received at different times due to network latency or other reasons, the aligned voice data will maintain the rhythm of the background music, thus providing a more synchronized chorus effect.

[0053] According to some implementations of this disclosure, during the period when a user joins the live broadcast room, network latency can be determined by sending test data, and this network latency can be used to perform alignment operations. Assuming that the start time of voice data 312 is mapped to time point T1 of standard time 330, and the start time of voice data 322 is mapped to time point T2 of standard time 330, then voice data 312 and voice data 322 can be aligned based on the time difference (T2-T1).

[0054] According to some implementations of this disclosure, supplementary enhancement information (SEI) can be inserted into the voice data stream as a basis for time synchronization. In the process of aligning first voice data and second voice data to generate second audio data, the first voice data can be parsed to determine first time information describing the time difference between the first voice data and background music (e.g., extracted from the SEI of the first voice data); the second voice data can be parsed to determine second time information describing the time difference between the second voice data and background music (e.g., extracted from the SEI of the second voice data); and based on the first and second time information, the first and second voice data are aligned to generate the second audio.

[0055] Using some implementations of this disclosure, vocal data from different audio users can be effectively aligned, thereby providing aligned vocal data to virtual chorus users. Although Figure 3 Two users are shown only schematically. Alternatively and / or additionally, there may be multiple users and the alignment operation described above can be performed on the voice data from each voice user in order to align the voice data of all voice users to the rhythm of the background music.

[0056] According to some implementations of this disclosure, an SEI (Sequence of Interchange) can be added to the second audio data to indicate the time difference between the second audio data and the background music. It should be understood that the terminal device may receive the second audio data with a delay; in this case, it is necessary to utilize the time information carried in the second audio data to align the rhythm of the vocal data in the second audio data with the rhythm of the background music. Furthermore, the terminal device can be instructed to align the second audio data and the background music based on the time information.

[0057] See Figure 4 Describe more information, this Figure 4 A block diagram 400 for aligning background music and second audio data is shown, according to some implementations of this disclosure. Figure 4 As shown, the length of the background music 410 can be greater than the length of the second audio data 220. SEI information can be extracted from the second audio data 220 to determine which time point in the background music 410 corresponds to the start time point of the second audio data 220. For example... Figure 4 As shown, the start time of the second audio data 220 corresponds to time point T3 in the background music 410. At this point, the background music 410 and the second audio data 222 can be aligned so that the second audio data 220 begins playing when the background music 410 is played at time point T3. It should be understood that although the alignment process has only been described above using the start time point as an example, multiple SEIs can be inserted into the second audio data 220 at predetermined intervals (e.g., 2 seconds or other time lengths) to indicate which frame in the background music the current audio data frame corresponds to. In this way, the playback of the background music and the second media data can be continuously adjusted at the terminal device to synchronize their rhythms.

[0058] According to some implementations of this disclosure, in order to improve user participation, better users can be selected from a large number of virtual chorus users and switched to audio users. Specifically, user evaluations can be determined, including at least one of the following: the user's vocal characteristics, rhythmic characteristics, stylistic characteristics, and network characteristics; and in response to determining that the evaluation meets predetermined switching conditions, the user is switched to an audio user in the chorus activity. Utilizing some implementations of this disclosure, the participation of different types of users can be increased, thereby promoting users to improve their singing skills and enhancing the overall chorus effect of the live stream.

[0059] Specifically, a user's voice characteristics can represent a score for the user's timbre itself. Rhythm characteristics can represent the degree of match between the rhythm of the user's singing and the rhythm of the background music of the song. Style characteristics can represent the degree of match between the user's singing style and the style of the song (e.g., rock, pop, classical, etc.). Network characteristics can represent the network status between the user's terminal device and the server device; for example, users with lower network latency can be preferentially selected as audio users, and so on. According to some implementations of this disclosure, the above characteristics can be normalized, and corresponding weights can be assigned to different characteristics. In this way, a more comprehensive assessment of whether a user is suitable to be selected as an audio user can be made. The evaluation can be determined at the user's terminal device and sent to the server device.

[0060] According to some implementations of this disclosure, in response to determining that the evaluation meets predetermined switching conditions, a user's type can be switched to another type different from the current type. The switching conditions can be set to multiple types; for example, one or more users with the lowest evaluation among multiple currently active users can be selected and converted into virtual chorus users. Alternatively and / or additionally, evaluations from all or some users can be received, and the N users with the highest evaluations can be selected as active users. It should be understood that, to ensure that the live broadcast effect does not change frequently, the switching action can be performed at predetermined time intervals (e.g., 30 seconds, or other values). According to some implementations of this disclosure, some users may withdraw from the chorus activity. If an active user withdraws from the chorus activity, a virtual chorus user can be switched to an active user; for example, the virtual chorus user with the highest evaluation can be switched to an active user.

[0061] The switching process from a regular audience user to a virtual chorus user has been described above. Alternatively and / or additionally, a virtual chorus user can be switched back to a regular audience user. A virtual chorus user can submit a request to withdraw from the chorus activity. Upon receiving this request, the user type and the type of audio data can be switched. According to some implementations of this disclosure, in response to receiving a user request to withdraw from the chorus activity, the provision of second audio data to the terminal device can be stopped; first audio data can be provided to the terminal device; and the terminal device can be instructed to release the player. At this time, there is no need to play local background music on the terminal device, so the local player can be released. Compared to existing solutions that require disconnecting the CDN connection between the terminal device and the live broadcast room and re-establishing the RTC connection, in this way, the server only needs to switch the audio data provided to the terminal device without performing other complex operations.

[0062] According to some implementations of this disclosure, background music during a choral activity can be retained on the terminal device until the activity ends. When a user participates in the choral activity, there is no need to resend the background music to the terminal device, thereby reducing network overhead and user waiting time. According to some implementations of this disclosure, in response to determining that the choral activity has ended, the terminal device can be instructed to delete the background music. It should be understood that since the choral activity has ended, the background music can be deleted directly to save storage space on the terminal device.

[0063] Based on some implementations of this disclosure, more user types can be set to meet the needs of different users in a more granular way. See also Figure 5 Describe more details about user types, the Figure 5 A block diagram 500 illustrating different users in a live streaming room according to some implementations of this disclosure is shown. For example... Figure 5 As shown, users with audio (510) can be set, and the upper limit for such users can be represented as N1 (e.g., N1 = 30 or other values). Users without audio (N2) can be set, and the upper limit for such users can be represented as N2 (e.g., N1 = 970 or other values). Here, the audio of user 510 will be used to generate chorus data and will be heard by other users, while the audio of user 520 will not be used to generate chorus data and will not be heard by other users.

[0064] According to some implementations of this disclosure, the audio user 510 and the silent user 520 can interact with the live broadcast room via an RTC connection. The silent user 520 can receive voice data from the audio user, play background music on its local terminal device, and simulate the auditory effect of participating in a chorus. For the silent user, the auditory effect of participating in a chorus can be simulated on the terminal device. The silent user can hear its own voice and the voices of other users, and therefore will think that it is participating in a chorus. However, the voice data of the silent user is not collected, so the silent user does not significantly increase the server's resource consumption.

[0065] According to some implementations of this disclosure, switching between silent and spoken users can be allowed. For example, a user with a higher rating can be selected as a spoken user, and other users can be selected as silent users. Using some implementations of this disclosure, even with a large number of silent users, the server workload can be kept within an acceptable range. This can significantly increase the number of users participating in choral activities, thus meeting the needs of more users. Since both types of users connect to the server via RTC, user switching does not result in re-establishing a connection; instead, it only requires re-determining the data source for collecting vocal data.

[0066] Furthermore, virtual chorus users 530 can be set. These users receive vocal data via a CDN connection, incurring no RTC-related overhead, and there is no upper limit to the number of users. Furthermore, switching rules between different types of users can be set. For example, user types can be set according to the order in which users request to participate in the chorus activity. Users 1 to N1 can be set as audio users 510, users N1+1 to N2 as silent users 520, and users N2+1 and beyond can be set as virtual chorus users. Alternatively and / or additionally, user types can be set based on evaluation, such as setting users with high evaluations as audio users, users with medium evaluations as silent users, and users with low evaluations as virtual chorus users. Alternatively and / or additionally, user types can be set based on other rules, such as users having priorities, and setting high-priority users as audio users, etc. Users can switch between different types; for example, after an audio user leaves the chorus activity, a user with a high evaluation can be selected from the silent users to become an audio user, etc.

[0067] By utilizing the implementation method of this disclosure, and distinguishing between different user types, only vocal data from users with voices can be collected during choral activities, and then this vocal data can be used to generate choral data. Since silent users do not cause significant resource overhead for the RTC service, the above method can increase the number of users participating in choral activities while meeting resource consumption requirements, and satisfy the need for more users to participate. Furthermore, virtual choral users do not consume RTC resources but only receive audio including vocal data; in this way, the number of users participating in choral activities can be further increased.

[0068] According to some implementations of this disclosure, users in certain choral activities can include lead singers and backing singers. Lead singers can participate in the entire chorus, while backing singers can sing only designated parts of the song. In this case, the lead singers are vocal users, and some backing singers are vocal users while others are silent users. The choral data can further include the vocal data of the lead singers participating in the chorus. By distinguishing between lead singers and backing singers using some implementations of this disclosure, the user management process can be further refined, thereby improving the management efficiency of the live stream. Alternatively and / or additionally, in a large chorus, vocal users can participate in the entire chorus, i.e., singing every line of lyrics. In this case, the distinction between lead singers and backing singers is not necessary.

[0069] The steps involved in managing choral activities have been described above. See below for further details. Figure 6 Describe the overall process of managing choral activities. Figure 6A block diagram 600 illustrating some implementations of this disclosure for participating in interactive activities is shown. For example... Figure 6 As shown, the live streaming room can involve chorus users (e.g., lead singer, vocal backing vocals, silent backing vocals, and virtual chorus users) and non-chorus users (e.g., guests and audience). A main chorus room service 610, a secondary chorus room service 612, and a post-processing service 614 can be established.

[0070] The main chorus room service 610 corresponds to the RTC room in a regular live stream, visible to all users. At this time, all users' vocal data is published to the main room, and all users can obtain complete chorus data from the main room. The secondary chorus room service 612 can be identified using the main room's identifier (e.g., roomID) + a fixed suffix (e.g., _aux), and is visible to the backing vocalists. The secondary room can provide single vocal data for the lead vocalist and facilitates flexible switching between the main room and the secondary room when backing vocalists join the chorus. The post-processing service 614 can align multiple data streams to a standard time and perform audio synthesis. Furthermore, the post-processing service 614 can have CDN publishing capabilities.

[0071] Lead singer A can play background music (BGM) on a local terminal device. The acquisition device can capture lead singer A's vocal data (represented by MIC A). The main chorus room service 610 can receive both MIC A and BGM. The secondary chorus room service 612 can receive vocal data (MIC A) from lead singer A. Optionally and / or additionally, the live stream may include guest E, who does not participate in the chorus. The main chorus room service 610 can provide all streams to guest E and receive vocal data (represented by MIC E) from guest E.

[0072] The chorus sub-room service 612 can receive vocal data MIC A from lead singer A. As shown by arrow 620, the chorus sub-room service 612 can send vocal data MIC A from lead singer A to vocal backing vocalist B, silent backing vocalist C, and vocal backing vocalist D respectively. As shown by arrows 621, 622, and 623, the chorus main room service 610 can send non-lead vocals to vocal backing vocalist B, silent backing vocalist C, and vocal backing vocalist D respectively. Each backing vocalist can pre-download and play background music (BGM) on their local terminal device and participate in the chorus activity along with the BGM and non-lead vocals.

[0073] At this point, the vocals of the vocal backing vocalists B and D are captured (MIC B and MIC D) and received by the main chorus room 610, while the vocal data of the other silent backing vocalist C is not sent to the main chorus room service 610, but is only played on the local terminal device. As shown by arrows 624 and 626, the vocal data MIC B and MIC D are pushed to the main chorus room service 610. As shown by arrow 625, the vocal data from the silent backing vocalist C is not pushed to the main chorus room 610. At this time, the server 616 can utilize the RTC connection to support more users, for example, it can support approximately 30 vocal backing vocalists and thousands of silent backing vocalists.

[0074] Furthermore, the post-processing service 614 can receive vocal data MICA from lead singer A, vocal data MICB and MICD from vocal backing singers B and D, vocal data MICE from guests, and background music (BGM) from the main chorus room service 610, and then align the multiple data streams to generate two CDN data streams. Specifically, CDN1 data (i.e., the first audio data) may include BGM and vocals, and CDN2 (i.e., the second audio data) may include only vocals. CDN1 can then be invoked to distribute CDN1 data to the audience, and CDN2 can be invoked to distribute CDN2 data to virtual chorus users. After the chorus ends, all backing singers and virtual chorus users can exit the main chorus room service 610.

[0075] By utilizing the implementation method disclosed herein, user participation in choral singing can be supported in a simpler and more efficient way by generating two different audio data. Furthermore, this method eliminates the need to set a limit on the number of users participating in choral activities, thereby supporting more users to participate in choral singing.

[0076] It should be understood that although the steps for participating in interactive activities have been described above using a choral activity as an example, alternatively and / or additionally, interactive activities may include recitation activities. Users can participate in recitation activities in a similar manner. For example, in a recitation activity with background music, a large number of users participating in the recitation activity can be divided into audio users and silent users. In this case, the user's voice data can only be heard by themselves and is not transmitted to other users. In this way, the auditory effect of participating in the recitation activity can be simulated locally at the terminal device. The user does not connect to the interactive service of the live broadcast room (e.g., real-time communication service) but receives audio data through a passive service (e.g., content delivery network service). In this way, it is not necessary to set an upper limit on the number of users participating in the recitation activity, thereby supporting more users to participate in the recitation.

[0077] Example process

[0078] Figure 7A flowchart of a method 700 for participating in an interactive activity according to some implementations of this disclosure is shown. At block 710, in response to determining that a user participating in the interactive activity is an audience type, first audio data is provided to the user's terminal device. The first audio data includes background music and voice data of the interactive activity, the voice data originating from at least one audio user participating in the interactive activity. At block 720, in response to receiving a user request from the user to participate in the interactive activity, the provision of the first audio data to the terminal device is stopped. At block 730, second audio data is provided to the terminal device, the second audio data including the voice data of the interactive activity.

[0079] According to some implementations of this disclosure, the method 700 further includes: providing background music for the chorus activity to the terminal device; and instructing the terminal device to start a player to play the background music.

[0080] According to some implementations of this disclosure, the second audio data includes time information describing the time difference between the second audio data and the background music, and the method 700 further includes: instructing the terminal device to align the second audio data and the background music based on the time information.

[0081] According to some implementations of this disclosure, providing second audio data includes: receiving first voice data from a first voice user among at least one voice user and second voice data from a second voice user among at least one voice user; and aligning the first voice data and the second voice data to generate second audio data.

[0082] According to some implementations of this disclosure, aligning first voice data and second voice data to generate second audio data includes: parsing the first voice data to determine first time information describing the time difference between the first voice data and background music; parsing the second voice data to determine second time information describing the time difference between the second voice data and background music; and aligning the first voice data and the second voice data based on the first time information and the second time information to generate second audio data.

[0083] According to some implementations of this disclosure, the method 700 further includes: in response to receiving a user request from a user to exit the chorus activity, stopping the provision of second audio data to the terminal device; providing first audio data to the terminal device; and instructing the terminal device to release the player.

[0084] According to some implementations of this disclosure, the method 700 further includes: instructing a terminal device to delete background music in response to determining that the chorus activity has ended.

[0085] According to some implementations of this disclosure, stopping the provision of the first audio data to the terminal device further includes: in response to receiving a user request from a user to participate in a chorus activity, determining the number of users with at least one voice; and in response to determining that the number of users has reached a predetermined threshold, stopping the provision of the first audio data to the terminal device.

[0086] According to some implementations of this disclosure, at least one terminal device of an audio user has a first service connection with a server providing interactive activities, and the user's terminal device has a second service connection with the server providing interactive activities.

[0087] According to some implementations of this disclosure, the method 700 further includes: determining a user's evaluation, the evaluation including at least one of the following: the user's voice characteristics, rhythm characteristics, style characteristics, and network characteristics; and in response to determining that the evaluation meets predetermined switching conditions, switching the user to a vocal user in the choral activity.

[0088] According to some implementations of this disclosure, interactive activities include at least one of the following: choral activities or recitation activities.

[0089] Example devices and equipment

[0090] Figure 8 A block diagram of an apparatus 800 for participating in an interactive activity according to some implementations of the present disclosure is shown. The apparatus 800 includes: a first providing module 810 configured to provide first audio data to a user's terminal device in response to determining that the user participating in the interactive activity is an audience type, the first audio data including background music and voice data of the interactive activity, the voice data being from at least one audio user participating in the interactive activity; a stopping module 820 configured to stop providing the first audio data to the terminal device in response to receiving a user request from the user participating in the interactive activity; and a second providing module 830 configured to provide second audio data to the terminal device, the second audio data including the voice data of the interactive activity.

[0091] According to some implementations of this disclosure, the device 800 further includes a processing module configured to: provide background music for a choral activity to a terminal device; and instruct the terminal device to start a player to play the background music.

[0092] According to some implementations of this disclosure, the second audio data includes time information describing the time difference between the second audio data and the background music, and the processing module is further configured to: instruct the terminal device to align the second audio data and the background music based on the time information.

[0093] According to some implementations of this disclosure, the second providing module is further configured to: receive first voice data from a first voice user among at least one voice user and second voice data from a second voice user among at least one voice user; and align the first voice data with the second voice data to generate second audio data.

[0094] According to some implementations of this disclosure, the second providing module is further configured to: parse the first voice data to determine first time information describing the time difference between the first voice data and the background music; parse the second voice data to determine second time information describing the time difference between the second voice data and the background music; and align the first voice data and the second voice data based on the first time information and the second time information to generate second audio data.

[0095] According to some implementations of this disclosure, the first providing module is further configured to: stop providing second audio data to the terminal device in response to receiving a user request from the user to exit the chorus activity; provide first audio data to the terminal device; and instruct the terminal device to release the player.

[0096] According to some implementations of this disclosure, the processing module is further configured to: instruct the terminal device to delete the background music in response to determining that the chorus activity has ended.

[0097] According to some implementations of this disclosure, the second providing module is further configured to: determine the number of users with at least one voice in response to receiving a user request from a user to participate in a chorus activity; and stop providing first audio data to the terminal device in response to determining that the number of users has reached a predetermined threshold.

[0098] According to some implementations of this disclosure, at least one terminal device of an audio user has a first service connection with a server providing interactive activities, and the user's terminal device has a second service connection with the server providing interactive activities.

[0099] According to some implementations of this disclosure, the module is further configured to: determine a user's evaluation, the evaluation including at least one of the following: the user's vocal characteristics, rhythmic characteristics, stylistic characteristics, and network characteristics; and, in response to determining that the evaluation meets predetermined switching conditions, switch the user to a vocal user in the choral activity.

[0100] According to some implementations of this disclosure, interactive activities include at least one of the following: choral activities or recitation activities.

[0101] Figure 9 A block diagram of a device 900 capable of implementing various implementations of the present disclosure is shown. It should be understood that... Figure 9The computing device 900 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementation described herein. Figure 9 The computing device 900 shown can be used to implement the method described above.

[0102] like Figure 9 As shown, computing device 900 is in the form of a general-purpose computing device. Components of computing device 900 may include, but are not limited to, one or more processors 910, memory 920, storage devices 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. Processor 910 may be a physical or virtual processor and is capable of performing various processes based on programs stored in memory 920. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 900.

[0103] Computing device 900 typically includes multiple computer storage media. Such media can be any available media accessible to computing device 900, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 920 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 930 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data (e.g., training data for training) and accessible within computing device 900.

[0104] The computing device 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 9 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 920 may include computer program product 925 having one or more program modules configured to perform various methods or actions of various implementations of this disclosure.

[0105] The communication unit 940 enables communication with other computing devices via a communication medium. Additionally, the components of the computing device 900 can function as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 900 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0106] Input device 950 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 960 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 900 can also communicate as needed with one or more external devices (not shown) via communication unit 940. These external devices, such as storage devices, display devices, etc., can communicate with one or more devices that enable user interaction with computing device 900, or with any device (e.g., network card, modem, etc.) that enables computing device 900 to communicate with one or more other computing devices. Such communication can be performed via input / output (I / O) interfaces (not shown).

[0107] According to an implementation of this disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the method described above. According to an implementation of this disclosure, a computer program product is provided, on which a computer program is stored, which, when executed by a processor, implements the method described above.

[0108] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0109] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0110] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0112] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for participating in an interactive activity, comprising: In response to determining that the user participating in the interactive activity is an audience type, first audio data is provided to the user's terminal device. The first audio data includes background music and voice data of the interactive activity, and the voice data comes from at least one voice user participating in the interactive activity. In response to receiving a user request from the user to participate in the interactive activity, Stop providing the first audio data to the terminal device; as well as Provide the terminal device with second audio data, the second audio data including the human voice data of the interactive activity.

2. The method according to claim 1, further comprising: Provide the background music for the interactive activity to the terminal device; as well as The terminal device is instructed to start the player to play the background music.

3. The method of claim 2, wherein the second audio data includes time information describing the time difference between the second audio data and the background music, and the method further includes: The terminal device is instructed to align the second audio data and the background music based on the time information.

4. The method of claim 1, wherein providing the second audio data comprises: Receive first voice data from a first voice user among the at least one voice users and second voice data from a second voice user among the at least one voice users; as well as Align the first voice data with the second voice data to generate the second audio data.

5. The method of claim 4, wherein aligning the first voice data with the second voice data to generate the second audio data comprises: The first voice data is analyzed to determine first time information describing the time difference between the first voice data and the background music; The second voice data is analyzed to determine second time information describing the time difference between the second voice data and the background music; as well as Based on the first time information and the second time information, the first voice data and the second voice data are aligned to generate the second audio data.

6. The method of claim 2, further comprising: In response to receiving a user request from the user to exit the interactive activity, Stop providing the second audio data to the terminal device; Provide first audio data to the terminal device; as well as Instruct the terminal device to release the player.

7. The method of claim 6, further comprising: In response to determining that the interactive activity has ended, the terminal device is instructed to delete the background music.

8. The method of claim 1, wherein stopping the provision of the first audio data to the terminal device further comprises: In response to receiving a user request from the user to participate in the interactive activity, the number of users of the at least one voice user is determined; as well as In response to determining that the number of users has reached a predetermined threshold, the first audio data is stopped being provided to the terminal device.

9. The method of claim 1, wherein a first service connection exists between the terminal device of the at least one voice user and the server providing the interactive activity, and a second service connection exists between the user's terminal device and the server providing the interactive activity.

10. The method of claim 1, further comprising: The evaluation of the user is determined, and the evaluation includes at least one of the following: the user's voice characteristics, rhythm characteristics, style characteristics, and network characteristics; as well as In response to determining that the evaluation meets the predetermined switching conditions, the user is switched to the voice user in the interactive activity.

11. The method of claim 1, wherein the interactive activity includes at least one of the following: a choral activity or a recitation activity.

12. A device for participating in interactive activities, comprising: A first providing module is configured to provide first audio data to the user's terminal device in response to determining that the user participating in the interactive activity is an audience type. The first audio data includes background music and voice data of the interactive activity, and the voice data comes from at least one voice user participating in the interactive activity. A stop module is configured to stop providing the first audio data to the terminal device in response to receiving a user request from the user to participate in the interactive activity. as well as A second providing module is configured to provide second audio data to the terminal device, the second audio data including the human voice data of the interactive activity.

13. An electronic device, comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 11 when executed by the at least one processor.

14. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 11.

15. A computer instruction product comprising computer instructions, wherein the computer instructions, when executed by a processor, implement the method according to any one of claims 1 to 11.

Images