Audio processing method, apparatus, device, and storage medium

Abstract

Embodiments of the present disclosure provide an audio processing method, apparatus, device and storage medium. The method comprises: determining at least one participant of a karaoke event joined in a live broadcast; determining at least one spatial parameter corresponding to each of the at least one participant based on layout information corresponding to the at least one participant, each of the at least one spatial parameter indicating a relative positional relationship between the corresponding participant and a reference listener of music; and obtaining a karaoke audio stream for the karaoke event, the karaoke audio stream being generated by processing at least one audio stream corresponding to the at least one participant using the at least one spatial parameter.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to audio processing methods, apparatus, devices, computer-readable storage media, and computer program products. Background Technology

[0002] With the rapid development of internet technology, live streaming has gradually become a new way of exchanging information. For example, in a live-streamed chorus event, multiple participants can sing the same song. The multiple audio streams from these participants can then be merged into a single chorus audio stream. This recreates the interactive experience of a real-life group singing session. Summary of the Invention

[0003] In a first aspect of this disclosure, an audio processing method is provided. The method includes: identifying at least one participant in a live chorus event; determining at least one spatial parameter corresponding to each of the at least one participant based on layout information corresponding to the at least one participant, the layout information indicating the layout of the at least one participant on a live chorus interface, each of the at least one spatial parameter indicating a relative positional relationship between the corresponding participant and a reference listener; and obtaining a chorus audio stream for the chorus event, the chorus audio stream being generated by processing at least one audio stream corresponding to the at least one participant using the at least one spatial parameter.

[0004] In a second aspect of this disclosure, an audio processing method is provided. The method includes: receiving at least one spatial parameter from a terminal device associated with the broadcaster of a live chorus event, the at least one spatial parameter indicating a relative positional relationship between at least one participant in the chorus event and a reference listener; receiving at least one audio stream corresponding to the chorus event from at least one participant; performing processing on the corresponding audio stream in the at least one audio stream based on each of the at least one spatial parameter to obtain at least one processed audio stream; and generating a chorus audio stream based at least on the at least one processed audio stream.

[0005] In a third aspect of this disclosure, an apparatus for audio processing is provided. The apparatus includes: a participant determination module configured to determine at least one participant joining a live chorus event; a parameter determination module configured to determine at least one spatial parameter corresponding to each of the at least one participant based on layout information corresponding to the at least one participant, the layout information indicating the layout of the at least one participant on a live chorus interface, each of the at least one spatial parameter indicating a relative positional relationship between the corresponding participant and a reference listener; and an audio stream acquisition module configured to acquire a chorus audio stream for the chorus event, the chorus audio stream being generated by processing at least one audio stream corresponding to the at least one participant using the at least one spatial parameter.

[0006] In a fourth aspect of this disclosure, an apparatus for audio processing is provided. The apparatus includes: a parameter receiving module configured to receive at least one spatial parameter from a terminal device associated with the broadcaster of a live chorus event, the at least one spatial parameter indicating a relative positional relationship between at least one participant in the chorus event and a reference listener; an audio stream receiving module configured to receive at least one audio stream corresponding to the chorus event from at least one participant; an audio stream processing module configured to perform processing on the corresponding audio stream of the at least one audio stream based on each of the at least one spatial parameter to obtain at least one processed audio stream; and an audio stream generation module configured to generate a chorus audio stream based at least on the at least one processed audio stream.

[0007] In a fifth aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the methods of the first or second aspect.

[0008] In a sixth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions that can be executed by a processor to implement the methods of the first or second aspect.

[0009] In a seventh aspect of this disclosure, a computer program product is provided, including computer-executable instructions, wherein when executed by a processor, the computer-executable instructions implement the method according to a first or second aspect of this disclosure.

[0010] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0012] Figure 1 A schematic diagram is shown of an example environment in which embodiments of the present disclosure may be implemented;

[0013] Figure 2 A flowchart illustrating an audio processing procedure according to some embodiments of the present disclosure is shown;

[0014] Figure 3 A flowchart of the signaling flow for audio processing according to some embodiments of this disclosure is shown;

[0015] Figure 4A and Figure 4B Schematic diagrams illustrating examples of layouts according to some embodiments of the present disclosure are shown respectively;

[0016] Figure 4C A schematic diagram illustrating an example of a layout interface according to some embodiments of the present disclosure is shown;

[0017] Figure 5 A schematic structural block diagram of an example apparatus for audio processing according to some embodiments of the present disclosure is shown;

[0018] Figure 6 A schematic structural block diagram of an example apparatus for audio processing according to other embodiments of the present disclosure is shown; and

[0019] Figure 7 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0020] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0021] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0022] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0023] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0024] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0025] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0026] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0028] As mentioned above, with the rapid development of internet technology, live streaming has gradually become a new way of information exchange. For example, in a live chorus event, multiple participants can sing the same song. The multiple audio streams from these participants can then be merged into a single chorus audio stream to recreate the interactive experience of a real-life group singing session. However, traditional live music chorus broadcasts directly merge multiple audio streams into a single chorus audio stream, which can easily lead to overlap between the voices of different participants, making it difficult to distinguish their voices and resulting in poor spatial awareness. Therefore, it is clear that the synthesis quality of chorus audio streams in live music chorus broadcasts still needs improvement.

[0029] In view of this, embodiments of the present disclosure propose an improved audio processing scheme. In this scheme, at least one participant in a live-streamed chorus event is identified. Based on layout information corresponding to the at least one participant, at least one spatial parameter corresponding to each participant is determined. The layout information can indicate the layout of the at least one participant on the live-streamed chorus interface. Each spatial parameter can indicate the relative positional relationship between the corresponding participant and a reference listener of the music. Then, using the at least one spatial parameter, at least one audio stream corresponding to the at least one participant is processed to obtain a chorus audio stream for the chorus event.

[0030] In the embodiments of this disclosure, spatial parameters are used to process the corresponding audio stream, which can form an audio stream with spatial audio effects. The choral audio stream formed by the merging of spatial audio effects can, to a certain extent, simulate the sound effect of a musical chorus performed in a choral formation corresponding to the corresponding layout, providing listeners with an immersive audiovisual experience. Furthermore, sound overlap is less likely to occur in the choral audio stream, allowing listeners to distinguish the voices of more participants and thus enabling more participants to engage in the choral event.

[0031] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0032] Example Environment

[0033] Figure 1A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In environment 100, one or more users 110-1, 110-2, 110-3, ..., 110-N can watch live streams and participate in live events through their respective associated terminal devices 120-1, 120-2, ..., 120-N. User 110 may also be referred to as a participant, for example. For ease of discussion, users 110-1, 110-2, ..., 110-N may be collectively referred to as user 110 or individually, and terminal devices 120-1, 120-2, ..., 120-N may be collectively referred to as terminal device 120 or individually. Among all users 110, one or more users, such as user 110-1, can establish live events and initiate live streams through associated terminal device 120-1.

[0034] In some scenarios, the user 110-1 who initiates the live stream can also be referred to as the host of the live stream event, a host participant, the live streamer, or the live stream administrator. One or more other users 110 may include guest participants 110-N, representing users who participate in the live stream interaction initiated by the host. In addition to guest participants in the live stream interaction, other users 110 may also include viewers, listeners, or viewers of the live stream event.

[0035] In some embodiments, the terminal device 120 may have an application installed that can provide live streaming services, or may have access to a website that can provide live streaming services. The user 110 can operate the terminal device 120 to access the corresponding application or website. Accordingly, the terminal device 120 may present a corresponding live streaming interface, which may provide live streaming content, such as audio live streaming content or video live streaming content.

[0036] In embodiments of this disclosure, environment 100 may include one or more servers 130-1, 130-2, ..., 130-N. In some embodiments, terminal device 120 may also communicate with servers 130-1, 130-2, ..., 130-N via network 132 to provide live streaming services. The one or more servers 130-1, 130-2, ..., 130-N may individually or collaboratively provide live streaming services to terminal device 120. In some embodiments, servers 130-1, 130-2, ..., 130-N may store data generated during live streaming events, including live interaction data generated during live streaming interactions.

[0037] Terminal device 120 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 120 may also support any type of user-facing interface (such as "wearable" circuitry). Server 130 can be any type of computing system / server capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0038] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0039] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0040] Example process

[0041] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure. Figure 2 A flowchart of an audio processing procedure 200 according to some embodiments of the present disclosure is shown. Part or all of the procedure 200 may be implemented by the terminal device 120, or by the terminal device 120 in cooperation with the server 130. In the following discussion, for ease of discussion, the execution of the procedure 200 is described from the perspective of the terminal device 120, but this is merely exemplary. In some examples, the terminal device 120 may be the broadcaster's terminal device. Of course, the terminal device 120 may also be the terminal device of other parameter parties in the chorus event; the embodiments of the present disclosure do not limit this.

[0042] In box 210, terminal device 120 determines at least one participant joining a live-streamed chorus event. This chorus event can be understood as an event involving a musical chorus performed live. The at least one participant can be understood as someone who participates in the musical chorus and whose corresponding audio stream will be used to synthesize the chorus audio stream. In some embodiments, the live stream may include a broadcaster and multiple viewers joining the live stream, any one of whom can choose to join the chorus event. The at least one participant can be selected from the multiple participants joining the chorus event, and the audio stream corresponding to the at least one participant will be used to synthesize the chorus audio stream.

[0043] As an example, server 130 can determine the scores of the audio streams of multiple participants. Server 130 can sort the multiple participants based on their scores and select a predetermined number of participants from the top of the sort. The audio streams of this predetermined number of participants will be used to synthesize a choral audio stream. Terminal device 120 can receive instructions (e.g., a list) from server 130 regarding this predetermined number of participants. In this way, audio streams with better sound quality can be selected from the participants joining the choral event for generating the choral audio stream.

[0044] It should be understood that the above-described strategy for determining the participants is merely exemplary. In practical applications, other appropriate methods can also be used to select participants for synthesizing the chorus audio stream from among multiple participants joining the chorus event. For example, terminal device 120 can also determine the at least one participant based on the broadcaster's selection. It should also be noted that during the live broadcast, the at least one participant may change dynamically. For example, server 130 can update the score of the participant's audio stream at predetermined intervals to update the at least one participant.

[0045] In some embodiments, terminal device 120 may determine at least one participant from a plurality of participants based on the respective roles of the participants joining the chorus event. Each participant's role indicates whether the participant is a lead vocalist or a backing vocalist. In some examples, terminal device 120 may select at least one participant corresponding to a backing vocalist role from a plurality of participants. Thus, subsequent processing can selectively be performed on the backing vocalist's audio stream.

[0046] As an example, Figure 3 A flowchart of a signaling stream 300 for audio processing according to some embodiments of the present disclosure is shown. The signaling stream 300 relates to server 130-1, server 130-2, terminal device 120-1, and terminal device 120-2. In some examples, terminal device 120-1 may be a terminal device associated with the broadcaster, and terminal device 120-2 may be a terminal device associated with other participants joining the chorus event.

[0047] like Figure 3As shown, server 130-1 can respond to a participant's (e.g., the broadcaster or viewer) selection of music (also known as a song request) by sending (302) information related to the selected music to terminal device 120-1. This music-related information may include instructions for the music (e.g., track name) and may also include instructions for the lead singer, such as an identifier for the lead singer. Terminal device 120-1 can determine (304) the participant corresponding to the lead singer based on this identifier. Server 130-1 can also determine a list of participants based on a predetermined strategy. This list indicates the participants corresponding to the lead singer and the participants corresponding to the backing vocalists. Server 130-1 can send (306) this list to terminal device 120-1. Terminal device 120-1 can determine (308) at least one participant corresponding to the backing vocalist based on this list.

[0048] In return process 200, at box 220, terminal device 120 determines at least one spatial parameter corresponding to each of the at least one participant based on the layout information corresponding to at least one participant. The layout information can indicate the layout of the at least one participant on the live chorus interface. This layout can indicate the chorus formation of the at least one participant. In some examples, the layout can indicate the relative positional relationship between the chorus formation of the at least one participant and a reference listener, such as relative distance or relative angle. The reference listener can be understood as a virtual listener used to assist in identifying the relative position between the chorus formation and the listener. The live chorus interface can be an interface presented during the live broadcast or an interface used to configure the arrangement of the chorus formation of the at least one participant. Each of the at least one spatial parameter indicates the relative positional relationship between the corresponding participant and the reference listener of the music.

[0049] In some embodiments, the reference listener may have default spatial coordinates, and the at least one spatial parameter may include at least one spatial coordinate corresponding to the at least one participant. Upon obtaining layout information corresponding to the at least one participant, the terminal device 120 may determine the layout of the at least one participant on the live chorus interface based on the layout information. The terminal device 120 may determine at least one spatial coordinate corresponding to the at least one participant based on this layout.

[0050] As an example, Figure 4A A schematic diagram of an example 400A of a first layout of a live chorus interface according to some embodiments of the present disclosure is shown. Figure 4AAs shown, terminal device 120 can pre-construct a spatial coordinate system for the live chorus interface. The X-axis can be defined as the forward / backward direction of the listener, the Y-axis as the left / right direction, and the Z-axis as the up / down direction. Terminal device 120 can determine the spatial coordinates of the listener as, for example, (0,0,0). Terminal device 120 can determine multiple positions used to arrange participants and their corresponding spatial coordinates within the live chorus interface, such as positions 412-1, 412-2, 422-1, 422-2, etc. These multiple positions can indicate the chorus formation for the chorus event. Terminal device 110 can determine the position of at least one participant within the live chorus interface and, based on the position of each participant, determine the corresponding spatial coordinates. Of course, in practical applications, other spatial parameters can also be used to indicate the relative positional relationship between participants and the listener, not just spatial coordinates.

[0051] The following combination Figures 4A to 4C This document provides an illustrative example of the layout for participants on the live chorus interface, but it should not be construed as limiting the layout to the layout shown below. In practical applications, any other appropriate layout can be selected based on actual needs.

[0052] In some embodiments, the layout on the live chorus interface may include a first layout. The first layout indicates at least one first arrangement sequence, which is sequentially moved away from the reference listener, and each first arrangement sequence includes at least one position arranged in a straight line. Each position may house one participant. As an example, such as... Figure 4A As shown, the first layout may include arrangement sequence 410 and arrangement sequence 420. Arrangement sequence 410 includes positions 412-1, 412-2, 412-3, and 412-4 arranged along a straight line. Arrangement sequence 420 includes positions 422-1, 422-2, 422-3, and 422-4 arranged along a straight line. Utilizing the first layout allows the sound of the subsequently generated choral audio stream to be more focused, enhancing the resonance of the sound. It should be understood that the first layout is not limited to including two arrangement sequences (also referred to as each row), but may include one arrangement sequence or three or more arrangement sequences. Each arrangement sequence is not limited to including four positions, but may include other numbers of positions. The embodiments of this disclosure do not specifically limit this.

[0053] In some embodiments, the layout on the live chorus interface may include a second layout. The second layout indicates at least one second arrangement sequence, which is sequentially moved away from the reference listener, and each second arrangement sequence includes at least one position arranged along an arc. As an example, Figure 4BA schematic diagram of an example 400B of a second layout of a live chorus interface according to some embodiments of the present disclosure is shown. Figure 4B As shown, the second layout may include an arrangement sequence 440, which may include positions 442-1, 442-2, 442-3, 442-4, and 442-5 arranged along an arc. The second layout may also indicate the relative distance and angle between the arrangement sequence 440 and the reference listener 430. Utilizing the second layout can enhance the surround sound effect of the subsequently generated choral audio stream. It should be understood that the above layout is merely exemplary. In practical applications, other predetermined musical choral layouts may also be used; for example, the arrangement sequence may be wavy, or it may include straight segments and curved segments, etc. The embodiments of this disclosure do not limit this.

[0054] In some embodiments, the broadcaster or other participants can flexibly create a music chorus layout according to actual needs. Specifically, if a layout request is received, the terminal device 120 can present a layout interface for at least one participant. The terminal device 120 can determine the layout of at least one participant based on the indication of the layout position of at least one participant in the layout interface.

[0055] As an example, Figure 4C A schematic diagram of an example 400C of an interface layout according to some embodiments of the present disclosure is shown. Figure 4C As shown, if a layout request is received, terminal device 120 can present the layout interface shown in Example 400C. The layout interface can present icons 452-1, 452-2, 452-3, 452-4, and 452-5 to identify the layout positions. The broadcaster can adjust the positions of the icons in the layout interface using terminal device 120 or an attached device of terminal device 120. Terminal device 120 can determine that it has received an instruction for the layout position of a participant in response to the adjustment operation of the icon positions in the layout interface. Based on the layout of multiple icons in the layout interface, the layout of at least one participant is determined. This layout can indicate that multiple participants can be arranged in the manner of icons 452-1, 452-2, 452-3, 452-4, and 452-5 to form a chorus queue. This layout can also indicate the relative distance and relative angle between multiple participants and the reference listener 430.

[0056] In some embodiments, the terminal device 120 can determine the layout of the at least one participant on the live chorus interface based on music-related information about the chorus event, thereby obtaining layout information corresponding to the at least one participant. This music-related information may include, but is not limited to, the structure, genre, singing style, song type, lyric classification, etc., of the music. In this way, a chorus layout matching the music can be selected.

[0057] Alternatively or additionally, the terminal device 120 can determine the layout of the at least one participant in the live chorus interface based on information related to the at least one participant, thereby obtaining layout information corresponding to the at least one participant. This information related to the at least one participant may include, but is not limited to, the number of the at least one participant, the acoustic characteristics of each participant (e.g., timbre, pitch, duration, intensity), etc. In this way, a chorus layout matching the at least one participant can be selected.

[0058] Alternatively or additionally, the terminal device 120 may also determine the layout of the at least one participant in the live chorus interface based on music-related information and information related to the at least one participant. This allows for the selection of a chorus layout that matches the music and the participants, which helps improve the quality of the subsequently generated chorus audio stream.

[0059] Alternatively or additionally, the terminal device 120 may present multiple layouts (also referred to as candidate layouts) for the user (e.g., the broadcaster) to choose from. If a selection of one of the multiple layouts is received, the terminal device 120 can determine the selected layout as the layout for that at least one participant in the live chorus interface. These multiple layouts may include a first layout, a second layout, or other predetermined layouts, and may also include layouts pre-created by the broadcaster. As an example, the broadcaster can choose a layout from predetermined layouts such as a first layout or a second layout, and the terminal device 120, based on the broadcaster's selection, determines the layout selected by the broadcaster as the layout corresponding to the currently played song. In this way, the broadcaster can independently define the layout corresponding to the chorus event.

[0060] It should be noted that the above-described layout determination method is merely exemplary. In practical applications, the layout of the at least one participant on the live chorus interface can be a preset layout (e.g., pre-set to the first chorus layout, the second chorus layout, or a music chorus layout created by the broadcaster). This layout can also be a layout pre-configured by the broadcaster in the live broadcast configuration information (e.g., a layout pre-selected by the broadcaster from the first or second layout). The layout can also be a layout set by the server 130. For example, the server 130 can determine the layout of the at least one participant on the live chorus interface according to predetermined rules. The terminal device 120 can determine the layout of the at least one participant on the live chorus interface based on the layout instructions from the server 130. The embodiments of this disclosure do not limit this.

[0061] In some embodiments, the terminal device 120 can determine whether the number of at least one participant exceeds a predetermined number. If it is determined that the number of at least one participant exceeds the predetermined number, the terminal device 120 determines at least one spatial parameter corresponding to each of the at least one participant based on layout information. If it is determined that the number of at least one participant does not exceed the predetermined number, the terminal device 120 may choose not to configure spatial parameters for the at least one participant. When the number of participants in the chorus is small, sound overlap is less likely to occur. Even without processing the audio streams of the participants, the voices of each participant can usually be well distinguished, saving system resource consumption. When the number of participants in the chorus is large, processing the audio streams of each participant can effectively avoid sound overlap and improve the quality of the subsequently generated choral audio stream.

[0062] As an example, such as Figure 3 As shown, terminal device 120-1 can determine whether the number of participants in the backing vocal role exceeds a predetermined number (e.g., 3) based on the list of participants. If it is determined that the number of participants in the backing vocal role exceeds the predetermined number, terminal device 120-1 can activate (310) spatial audio mode. Then, in spatial audio mode, terminal device 120-1 can determine the layout of the participants in the backing vocal role on the live chorus interface, and based on the determined layout, determine (312) the spatial parameters of each participant in the backing vocal role. It should be understood that the above method for determining spatial parameters is only exemplary, and any other appropriate method can be selected to determine the spatial parameters of the participants. The embodiments of this disclosure do not limit the method for determining spatial parameters.

[0063] In return process 200, at box 230, terminal device 120 obtains a choral audio stream for the choral event, the choral audio stream being generated by processing at least one audio stream corresponding to at least one participant using at least one spatial parameter.

[0064] It should be noted that the generation process of the chorus audio stream (also known as merging) can be implemented on the terminal device associated with the broadcaster of the chorus event (e.g., terminal device 120-1), or on the server 130. The following will combine... Figure 3 The process of generating a choral audio stream is introduced using server 130-2 as an example.

[0065] In signaling stream 300, terminal device 120-1 sends a processing request (314) to server 130-2 associated with the chorus event. This processing request includes at least one spatial parameter. In some embodiments, terminal device 120-1 may be deployed with a Real-Time Communication Engine (RTCSDK), through which it can send the processing request to server 130-2.

[0066] In signaling stream 300, server 130-2 receives (316) the audio stream corresponding to the chorus event from the participating party's terminal device 120-2. It should be noted that, although... Figure 3 Only the terminal device 120-2 of the participating party is shown in the diagram, but in actual applications, server 130-2 can receive audio streams from the terminal devices of multiple participating parties. In the case of a broadcaster participating in a chorus, server 130-2 can also receive audio streams from the broadcaster's terminal device 120-1.

[0067] In signaling stream 300, server 130-2 performs (318) processing (also referred to as spatialization processing) on ​​the corresponding audio stream in at least one audio stream based on each of the at least one spatial parameter to obtain at least one spatialized audio stream. Then, server 130-2 generates (320) a chorus audio stream based at least on the at least one spatialized audio stream. In some embodiments, the spatial parameter may include spatial coordinates. Server 130-2 may perform spatialization processing on the corresponding audio stream based on each spatial coordinate to obtain a spatialized audio stream.

[0068] In some embodiments, the at least one participant corresponds to a backing vocalist in the music. Server 130-2 can receive an audio stream from the participant corresponding to the lead vocalist in the music. Then, server 130-2 can generate a chorus audio stream based on at least one spatially processed audio stream and the audio stream corresponding to the lead vocalist in the music. This allows for relatively good preservation of the original audio characteristics of the lead vocalist's audio stream, or allows for personalized processing of the lead vocalist's audio stream, which helps to highlight the lead vocalist's voice in the chorus audio stream. As an example, assuming the broadcaster is the lead vocalist, server 130-2 can receive the lead vocalist's audio stream from terminal device 120-1, and server 130-2 can also receive the backing vocalist's audio stream from terminal device 120-2. Server 130-2 can perform spatial processing on the backing vocalist's audio stream based on spatial coordinates. Then, based on the lead vocalist's audio stream and the spatially processed backing vocalist's audio stream, a chorus audio stream is generated.

[0069] In signaling stream 300, terminal device 120-1 can receive (322) the chorus audio stream from server 130-2. Server 130-2 can also push (324) the chorus audio stream to terminal device 120-2. Of course, server 130-2 can also push the chorus audio stream to other viewers of the live broadcast. Performing spatial processing and merging operations on the audio stream on server 130-2 can reduce the system resource overhead of the broadcaster's terminal device 120-1 and improve the processing efficiency of the audio stream.

[0070] In some embodiments, the merging operation of the choral audio streams can be configured to be performed by a terminal device 120. The terminal device 120 can receive at least one audio stream corresponding to at least one participant. The terminal device 120 can perform spatialization processing on the corresponding audio streams in the at least one audio stream based on each of the at least one spatial parameter to obtain at least one spatialized audio stream. Then, the terminal device 120 generates a choral audio stream by merging at least one spatialized audio stream.

[0071] As an example, terminal device 120-1 can receive at least one audio stream from the at least one participant from server 130-1 or server 130-2. Terminal device 120-1 can perform spatialization processing on the corresponding audio stream based on the spatial coordinates of each participant to obtain a spatialized audio stream. Then, terminal device 120-1 can generate a chorus audio stream based on the spatialized audio stream. After generating the chorus audio stream, terminal device 120-1 can push the chorus audio stream to server 130-2, which in turn pushes the chorus audio stream to each viewer.

[0072] In some embodiments, at least one participant corresponds to a backing vocalist in the music. Terminal device 120 receives an audio stream from the participant corresponding to the lead vocalist. Then, terminal device 120 generates a chorus audio stream based on the spatially processed at least one audio stream and the audio stream corresponding to the lead vocalist. This approach can relatively well preserve the original audio characteristics of the lead vocalist's audio stream, which is beneficial for highlighting the lead vocalist's voice in the chorus audio stream.

[0073] It should be noted that during a choral performance, the participants may change dynamically, and the spatialization of the audio stream and the generation of the synthesized audio stream may be ongoing. Therefore, the above process may be repeated multiple times during the choral performance.

[0074] In this way, in the embodiments of this disclosure, spatial parameters are used to process the corresponding audio stream, thereby forming an audio stream with spatial audio effects. The choral audio stream formed by the merging of spatial audio effects can, to a certain extent, simulate the sound effect of a musical chorus performed in a choral formation corresponding to the layout, providing listeners with an immersive audiovisual experience. Furthermore, sound overlap is less likely to occur in the choral audio stream, allowing listeners to distinguish the voices of more participants and thus enabling more participants to engage in the choral event.

[0075] Example devices and equipment

[0076] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 5 A schematic structural block diagram of an example device 500 for audio processing according to certain embodiments of the present disclosure is shown. Device 500 may be implemented as or included in terminal device 120. Various modules / components in device 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0077] like Figure 5 As shown, the apparatus 500 includes: a participant determination module 510 configured to determine at least one participant joining a live chorus event; a parameter determination module 520 configured to determine at least one spatial parameter corresponding to each of the at least one participant based on layout information corresponding to the at least one participant, the layout information indicating the layout of the at least one participant on the live chorus interface, and each of the at least one spatial parameter indicating the relative positional relationship between the corresponding participant and a reference listener; and an audio stream acquisition module 530 configured to acquire a chorus audio stream for the chorus event, the chorus audio stream being generated by processing at least one audio stream corresponding to at least one participant using at least one spatial parameter.

[0078] In some embodiments, the participant determination module 510 is further configured to: determine at least one participant from among the multiple participants based on their respective roles in the chorus event, wherein the role of each participant indicates whether the participant is a lead singer or a backing singer in the chorus event.

[0079] In some embodiments, the parameter determination module 520 is further configured to: determine whether the number of at least one participant exceeds a predetermined number; and in response to determining that the number of at least one participant exceeds the predetermined number, determine at least one spatial parameter corresponding to each of the at least one participant based on layout information.

[0080] In some embodiments, the layout information is determined based on at least one of the following: information related to the music of the choral event, information related to at least one participant, and selection of a layout among multiple layouts.

[0081] In some embodiments, the apparatus 500 further includes: a layout information determination module, configured to present a layout interface for at least one participant in response to a layout request; and to determine the layout of at least one participant on the live chorus interface based on an indication of the layout position of at least one participant in the layout interface, so as to obtain layout information.

[0082] In some embodiments, the layout includes one of the following: a first layout indicating at least one first arrangement sequence, the at least one first arrangement sequence being moved away from the reference listener in sequence, each first arrangement sequence including at least one position arranged along a straight line; and a second layout indicating at least one second arrangement sequence, the at least one second arrangement sequence being moved away from the reference listener in sequence, each second arrangement sequence including at least one position arranged along an arc.

[0083] In some embodiments, the apparatus 500 is implemented as or included in a terminal device associated with the broadcaster of the chorus event, and the audio stream acquisition module 530 is further configured to: send a processing request to a server associated with the chorus event, the processing request including at least one spatial parameter; and receive the chorus audio stream from the server.

[0084] In some embodiments, the apparatus 500 is implemented as or included in a terminal device associated with the broadcaster of the chorus event, and the audio stream acquisition module 530 is further configured to: receive at least one audio stream corresponding to at least one participant; perform processing on the corresponding audio stream in the at least one audio stream based on each of the at least one spatial parameters to obtain at least one processed audio stream; and generate a chorus audio stream by at least merging the at least one processed audio stream.

[0085] In some embodiments, at least one participant corresponds to a backing vocalist in a chorus event, and the audio stream acquisition module 530 is further configured to: receive an audio stream from the participant corresponding to the lead vocalist in the chorus event; and generate a chorus audio stream based on at least one processed audio stream and the audio stream corresponding to the lead vocalist.

[0086] Figure 6 A schematic structural block diagram of an example device 600 for audio processing according to certain embodiments of the present disclosure is shown. Device 600 may be implemented as or included in server 130. Various modules / components in device 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0087] like Figure 6 As shown, the apparatus 600 includes: a parameter receiving module 610 configured to receive at least one spatial parameter from a terminal device associated with the broadcaster of a live chorus event, wherein the at least one spatial parameter indicates the relative positional relationship between at least one participant in the chorus event and a reference listener; an audio stream receiving module 620 configured to receive at least one audio stream corresponding to the chorus event from at least one participant; an audio stream processing module 630 configured to perform processing on the corresponding audio stream of the at least one audio stream based on each of the at least one spatial parameter to obtain at least one processed audio stream; and an audio stream generation module 640 configured to generate a chorus audio stream based at least on the at least one processed audio stream.

[0088] In some embodiments, at least one participant corresponds to a backing vocalist in the chorus event, and the audio stream generation module 640 is further configured to: receive an audio stream from the participant corresponding to the lead vocalist in the chorus event; and generate the chorus audio stream based on the processed at least one audio stream and the audio stream corresponding to the lead vocalist in the chorus event.

[0089] In some embodiments, the device 600 further includes an audio stream push module configured to push the chorus audio stream to a viewer of the chorus event.

[0090] The units and / or modules included in devices 500 and 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and / or modules in devices 500 and 600 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0091] Figure 7 A block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 7 The electronic device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 7 The illustrated electronic device 700 may include or be implemented as Figure 1 Terminal device 120 or server 130, Figure 5 Device 500, or Figure 6 Device 600 in the middle.

[0092] like Figure 7 As shown, electronic device 700 is in the form of a general-purpose electronic device. Components of electronic device 700 may include, but are not limited to, one or more processors 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processor 710 may be a physical or virtual processor and is capable of performing various processes according to executable instructions stored in memory 720. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 700.

[0093] Electronic device 700 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 700.

[0094] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more executable instruction modules configured to perform various methods or actions of various embodiments of this disclosure.

[0095] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0096] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0097] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0098] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable and executable instructions.

[0099] These computer-executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-executable instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0100] Computer-executable instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, executable instruction, or portion of instructions, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0102] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. An audio processing method, comprising: Identify at least one party to join the live-streamed chorus event; Based on the layout information corresponding to the at least one participant, at least one spatial parameter corresponding to each of the at least one participant is determined. The layout information indicates the layout of the at least one participant on the live chorus interface. Each of the at least one spatial parameter indicates the relative positional relationship between the corresponding participant and the reference listener. as well as Obtain a choral audio stream for the choral event, the choral audio stream being generated by processing at least one audio stream corresponding to the at least one participant using the at least one spatial parameter.

2. The method of claim 1, wherein determining the at least one participant comprises: Based on the roles of the multiple participants joining the chorus event, at least one participant is determined from the multiple participants, wherein the role of each participant indicates that the participant is a lead singer or a backing singer in the chorus event.

3. The method according to claim 1, wherein determining at least one spatial parameter corresponding to each of the at least one participating party comprises: Determine whether the number of the at least one participating party exceeds a predetermined number; as well as In response to determining that the number of the at least one participant exceeds a predetermined number, the at least one spatial parameter corresponding to each of the at least one participant is determined based on the layout information.

4. The method of claim 1, wherein the layout information is determined based on at least one of the following: Information related to the music of the aforementioned choral event. Information related to the at least one participating party, Selection of the layout among multiple layouts.

5. The method according to claim 1, wherein the layout information is determined in the following manner: In response to a layout request, a layout interface for the at least one participant is presented; and Based on the indication of the arrangement position of the at least one participant in the arrangement interface, the layout of the at least one participant on the live chorus interface is determined to obtain the layout information.

6. The method of claim 1, wherein the layout comprises one of the following: A first layout, indicating at least one first arrangement sequence, the at least one first arrangement sequence being sequentially moved away from the reference listening side, each first arrangement sequence including at least one position arranged in a straight line. A second layout, the second layout indicating at least one second arrangement sequence, the at least one second arrangement sequence being sequentially moved away from the reference listener, each second arrangement sequence including at least one position arranged along an arc.

7. The method of claim 1, wherein the method is implemented on a terminal device associated with the broadcaster of the chorus event, and wherein obtaining the chorus audio stream for the chorus event comprises: A processing request is sent to the server associated with the chorus event, the processing request including at least the at least one spatial parameter; as well as Receive the chorus audio stream from the server.

8. The method of claim 1, wherein the method is implemented on a terminal device associated with the broadcaster of the chorus event, and wherein obtaining the chorus audio stream for the chorus event comprises: Receive the at least one audio stream corresponding to the at least one participant; Based on each of the at least one spatial parameters, the corresponding audio stream in the at least one audio stream is processed to obtain at least one processed audio stream; as well as The chorus audio stream is generated by merging at least one of the processed audio streams.

9. The method of claim 8, wherein the at least one participant corresponds to a backing vocalist in the chorus event, and wherein generating the chorus audio stream comprises: Receive audio streams from the participants corresponding to the lead singer in the chorus event; as well as The chorus audio stream is generated based on at least one processed audio stream and the audio stream corresponding to the lead singer role.

10. An audio processing method, implemented at a server, the method comprising: At least one spatial parameter is received from the terminal device associated with the broadcaster of the live chorus event, the at least one spatial parameter indicating the relative positional relationship between at least one participant in the chorus event and a reference listener; Receive at least one audio stream corresponding to the chorus event from the at least one participant; Based on each of the at least one spatial parameters, the corresponding audio stream in the at least one audio stream is processed to obtain at least one processed audio stream; as well as A chorus audio stream is generated based on at least one of the processed audio streams.

11. The method of claim 10, wherein the at least one participant corresponds to a backing vocalist in the chorus event, and wherein generating the chorus audio stream comprises: Receive audio streams from the participants corresponding to the lead singer in the chorus event; as well as The chorus audio stream is generated based on at least one processed audio stream and the audio stream corresponding to the lead singer in the chorus event.

12. The method of claim 10, further comprising: The chorus audio stream is pushed to the viewer of the chorus event.

13. An apparatus for audio processing, comprising: The participant identification module is configured to identify at least one participant joining the chorus event in the live stream; The parameter determination module is configured to determine at least one spatial parameter corresponding to each of the at least one participant based on the layout information corresponding to the at least one participant. The layout information indicates the layout of the at least one participant on the live chorus interface, and each of the at least one spatial parameters indicates the relative positional relationship between the corresponding participant and the reference listener. as well as An audio stream acquisition module is configured to acquire a choral audio stream for the choral event, the choral audio stream being generated by processing at least one audio stream corresponding to the at least one participant using the at least one spatial parameter.

14. An apparatus for audio processing, comprising: The parameter receiving module is configured to receive at least one spatial parameter from a terminal device associated with the broadcaster of a live chorus event, wherein the at least one spatial parameter indicates the relative positional relationship between at least one participant in the chorus event and a reference listener. An audio stream receiving module is configured to receive at least one audio stream corresponding to the chorus event from the at least one participant; An audio stream processing module is configured to perform processing on the corresponding audio stream in the at least one audio stream based on each of the at least one spatial parameters, so as to obtain at least one processed audio stream. as well as An audio stream generation module is configured to generate a chorus audio stream based on at least one of the processed audio streams.

15. An electronic device comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions, when executed by the at least one processor, causing the electronic device to perform the method according to any one of claims 1 to 9 or 10 to 12.

16. A computer-readable storage medium having stored thereon computer-executable instructions that can be executed by a processor to implement the method according to any one of claims 1 to 9 or 10 to 12.

17. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 9 or 10 to 12.

Images