An audio processing method, apparatus, device and storage medium

Abstract

Embodiments of the present application provide an audio processing method, device and equipment, and a storage medium, relating to the technical field of multimedia, which comprises: a first participant triggering a chorus participation operation in a target application in a director role, so that the target application displays a first interface of a chorus virtual room under the director role. In this way, the first participant can experience the interactive effect under the new role. Secondly, in response to the first participant, the original mixed audio of at least one singing object group is adjusted in response to an audio adjustment operation triggered by the first participant, and the obtained at least one target mixed audio is mixed with the original mixed audio of other singing object groups to obtain chorus audio. In this way, the first participant can adjust the mixed audio of each singing object group according to the actual situation, avoid low-quality audio signals from becoming the core sound, effectively improve the quality of the chorus audio, and further improve the chorus effect.

Description

Technical Field

[0001] This application relates to the field of multimedia technology, and in particular to an audio processing method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of computer and internet technologies, various applications with audio-visual functions have emerged. Participants can use these applications to experience audio-visual entertainment anytime, anywhere on the internet, and interact with other participants in different geographical locations.

[0003] With the relevant technology, online chorus is an interactive entertainment method that allows participants from different geographical locations to sing a song together. During the chorus, each participant listens to the accompaniment played on their local terminal and sings along with the other participants; the local terminal collects the audio signal and sends it to a mixing server via the internet. The mixing server mixes the received multiple audio signals to obtain a mixed signal, which is then returned to each terminal for playback.

[0004] However, in actual choral singing, the singing quality of different participants may vary, resulting in different quality audio signals collected from each participant. In this situation, directly mixing the audio signals from different participants may lead to some loud but low-quality audio signals becoming the core sound parts of the mixed signal, resulting in a poor mixed signal quality and thus affecting the overall choral effect. Secondly, in actual choral singing, there is a lack of interaction between the participants, which also affects the choral effect. Summary of the Invention

[0005] This application provides an audio processing method, apparatus, device, and storage medium to improve the quality of choral audio, thereby enhancing the choral effect and interactivity.

[0006] On one hand, embodiments of this application provide an audio processing method, the method comprising:

[0007] In response to the first participating object, the choral participation operation triggered by the command role in the target application is displayed, and the first interface of the choral virtual room is displayed under the command role. The first interface includes: multiple singing object groups, each singing object group including: at least one second participating object that accesses the choral virtual room as a singing role.

[0008] In response to the first participating object, for an audio adjustment operation triggered for at least one of the singing object groups, the chorus audio is played in the first interface; wherein, the chorus audio is obtained by adjusting the original mixed audio of each of the at least one singing object group, and then mixing the obtained target mixed audio with the original mixed audio of each of the other singing object groups.

[0009] On one hand, embodiments of this application provide an audio processing apparatus, the apparatus comprising:

[0010] The display module is used to respond to the first participating object and to display the first interface of the chorus virtual room under the commanding role in the target application in response to the chorus participation operation triggered by the commanding role. The first interface includes: multiple singing object groups, each singing object group including: at least one second participating object that accesses the chorus virtual room as a singing role.

[0011] The display module is further configured to respond to the first participating object and, in response to an audio adjustment operation triggered for at least one of the singing object groups, play choral audio in the first interface; wherein, the choral audio is obtained by adjusting the original mixed audio of each of the at least one singing object group separately, and then mixing the obtained target mixed audio with the original mixed audio of each of the other singing object groups.

[0012] Optionally, a settings module may also be included;

[0013] The setting module is specifically used for:

[0014] In response to the role selection operation triggered by the first participating object in the target application, the role of the first participating object is set to the command role.

[0015] Optionally, the setting module is further configured to:

[0016] In response to the role-switching operation triggered by the first participant in the first interface, the role of the first participant is switched to the singing role, and the second interface of the chorus virtual room is displayed under the singing role; or,

[0017] In response to the role switching operation triggered by the first participant in the first interface, the role of the first participant is switched to the audience role, and the third interface of the chorus virtual room under the audience role is displayed.

[0018] Optionally, the multiple groups of singing objects are divided according to a preset first grouping pattern;

[0019] The setting module is also used for:

[0020] In response to a mode selection operation triggered by the first participating object in the target application, the singing group mode is set to the first group mode.

[0021] Optionally, the setting module is specifically used for:

[0022] For each second participant who accesses the virtual chorus room as a singing role, the following steps are performed: extracting the vocal features of the second participant from its historical singing data; and obtaining the group category corresponding to the second participant in the first grouping mode based on the vocal features of the second participant.

[0023] Based on the grouping categories of the multiple second participants who access the virtual chorus room, the multiple second participants are divided into multiple singing object groups in the first grouping mode.

[0024] Optionally, the setting module is further configured to:

[0025] In response to the mode switching operation triggered by the first participating object in the target application, the singing group mode is switched to the second group mode, and multiple singing object groups in the second group mode are displayed in the first interface.

[0026] Optionally, it may also include a processing module;

[0027] The processing module is specifically used for:

[0028] Obtain the body movements of the first participating object;

[0029] Based on the physical movements, at least one audio adjustment instruction is obtained for each of the singing groups;

[0030] According to the audio adjustment instructions corresponding to each singing object group, the original mixed audio corresponding to each singing object group is adjusted to obtain the corresponding target mixed audio;

[0031] The obtained target mixed audio is mixed with the original mixed audio of each of the other singing groups to obtain the chorus audio.

[0032] Optionally, the body movements include: hand gestures;

[0033] The processing module is specifically used for:

[0034] Collect a video segment of the first participating object, the video segment comprising multiple frame images;

[0035] Identify at least one set of joints for each hand in each frame image;

[0036] Perform the following steps for each hand:

[0037] Based on the obtained set of multiple joint points of a hand, the motion features of the hand are extracted;

[0038] Based on the motion characteristics of the hand, the hand gesture is obtained.

[0039] Optionally, the processing module is specifically used for:

[0040] For at least one of the frame images, the following steps are performed: based on the set of joints of the hand in the frame image and the torso region identified from the frame image, the positional relationship between the hand and the torso region is obtained;

[0041] Based on at least one of the obtained positional relationships, at least one group of the singing objects controlled by the hand is determined;

[0042] Based on the hand gesture, query the preset gesture mapping data to obtain audio adjustment instructions for at least one of the singing object groups.

[0043] Optionally, the body movements include: lip movements;

[0044] The processing module is specifically used for:

[0045] Collect a video segment of the first participating object, the video segment comprising multiple frame images;

[0046] The lip region is segmented from each of the frame images;

[0047] Based on the segmented multiple lip regions, lip movement features are extracted;

[0048] Based on the lip movement characteristics, mouth shape actions are obtained.

[0049] Optionally, the processing module is specifically used for:

[0050] Based on the lip movements, preset lip mapping data is queried to obtain audio adjustment instructions for at least one of the singing object groups.

[0051] On one hand, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the aforementioned audio processing method, specifically including:

[0052] In response to a first participating object, the processor displays a first interface of the chorus virtual room under the commanding role, in order to direct the chorus participation operation triggered by the commanding role in the target application. The first interface includes: multiple singing object groups, each singing object group including: at least one second participating object that accesses the chorus virtual room as a singing role.

[0053] In response to the first participating object, the processor plays choral audio in the first interface for an audio adjustment operation triggered by at least one of the singing object groups; wherein, the choral audio is obtained by adjusting the original mixed audio of each of the at least one singing object group separately, and then mixing the obtained at least one target mixed audio with the original mixed audio of each of the other singing object groups; the choral audio is stored in memory.

[0054] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the above-described audio processing method.

[0055] On one hand, embodiments of this application provide a computer program product, the computer program product including a computer program stored on a computer-readable storage medium, the computer program including program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described audio processing method.

[0056] In this embodiment of the application, the first participant triggers the chorus participation operation in the target application as a conductor, so that the target application displays the first interface of the chorus virtual room under the conductor's role. In this way, the first participant can participate in the chorus interaction in a brand-new role as a conductor, so that the participant can experience different interactive effects and auditory experience.

[0057] Secondly, in response to the first participating object, for the audio adjustment operation triggered by at least one singing object group, the original mixed audio of each of the at least one singing object group is adjusted separately, and the obtained target mixed audio is mixed with the original mixed audio of each of the other singing object groups to obtain the chorus audio. In this way, the first participating object can adjust the mixed audio of each singing object group according to the actual situation, effectively avoiding some audio signals with large volume but low quality from becoming the core sound part of the chorus audio, thereby effectively improving the quality of the chorus audio and thus improving the chorus effect.

[0058] In addition, the first participants can adjust the unique choral audio according to their own preferences, which can make the content of the choral audio richer and more interesting, while increasing the fun and participation of the participants in the choral interaction. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 A schematic diagram of a system architecture provided in an embodiment of this application;

[0061] Figure 2 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0062] Figure 3 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0063] Figure 4 A flowchart illustrating an audio processing method provided in an embodiment of this application;

[0064] Figure 5 A schematic diagram of a first interface provided in an embodiment of this application;

[0065] Figure 6 A flowchart illustrating a role setting method provided in an embodiment of this application;

[0066] Figure 7 A flowchart illustrating a role setting method provided in an embodiment of this application;

[0067] Figure 8A A schematic diagram of a second interface provided in an embodiment of this application;

[0068] Figure 8B A schematic diagram of a second interface provided in an embodiment of this application;

[0069] Figure 9 A flowchart illustrating a role setting method provided in an embodiment of this application;

[0070] Figure 10A A schematic diagram of a second interface provided in an embodiment of this application;

[0071] Figure 10B A schematic diagram of a second interface provided in an embodiment of this application;

[0072] Figure 11 A schematic flowchart illustrating a mode setting method provided in an embodiment of this application;

[0073] Figure 12A schematic flowchart illustrating a mode setting method provided in an embodiment of this application;

[0074] Figure 13 A schematic diagram of a hand joint provided in an embodiment of this application;

[0075] Figure 14 This is a schematic diagram of gesture mapping data provided in an embodiment of this application;

[0076] Figure 15 This is a schematic diagram of lip shape mapping data provided in an embodiment of this application;

[0077] Figure 16 A flowchart illustrating an audio processing method provided in an embodiment of this application;

[0078] Figure 17 This is a schematic diagram of the structure of an audio processing device provided in an embodiment of this application;

[0079] Figure 18 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0080] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0081] For ease of understanding, the terms used in the embodiments of this invention are explained below.

[0082] Karaoke apps are music entertainment software installed on mobile devices. Users select songs from the software's extensive song library and then listen to the background music via the device's playback capabilities. Users then record and sing along to the music, creating a vocal performance. The app allows for multi-person duets and performance sharing via real-time or offline audio data transmission over the internet. Karaoke apps also offer features such as singing scoring, sound effects processing, audio enhancement, and community interaction, allowing users to enjoy the singing experience while improving their singing skills.

[0083] Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0084] Long Short-Term Memory (LSTM) networks are a special type of recurrent neural network (RNN) that addresses the vanishing gradient problem in long-sequence data processing by introducing memory units (cell states). LSTMs can capture dependencies in long sequences and are well-suited for deep learning models that handle time series, natural language processing, and speech recognition tasks.

[0085] Framing: Sound signals are continuous, while computers process discrete data. To enable computers to process and analyze sound signals, continuous sound signals are divided into multiple small segments, each segment being called a frame (i.e., an audio frame signal).

[0086] Windowing: Process each frame signal to gradually reduce the edge signal of the frame to zero, thereby reducing the jumps between frames. This makes the connection of each frame smoother and avoids noise caused by abrupt changes.

[0087] Mel-frequency cepstrum coefficients (MFCCs) are a feature extraction method used in speech signal processing and audio analysis, widely applied in speech recognition, speaker recognition, and music information retrieval. They simulate the nonlinear perception of sound frequencies by the human ear, effectively capturing key features of speech.

[0088] Fundamental frequency (F0) is the frequency of periodic vibrations in a speech signal, typically corresponding to the vibration frequency of the vocal cords. The fundamental frequency directly reflects the pitch information of speech and is important for emotional and musical analysis of speech.

[0089] Formants are energy-concentrated regions in the speech signal spectrum, corresponding to the resonant frequencies of the vocal tract. They are typically represented by F1, F2, and F3. Formant features reflect the shape and size of the vocal tract and are important for timbre and vowel recognition in speech.

[0090] MediaPipe Pose is a real-time human pose estimation solution based on machine learning. It can detect key points (such as joints and limbs) of the human body from images or videos and generate a human pose skeleton. MediaPipe Pose is efficient, lightweight, and cross-platform, suitable for mobile devices, desktop devices, and embedded devices.

[0091] MediaPipe Hands is a lightweight hand pose estimation model that can detect 21 key points of the hand (including finger joints and palm) and generate the hand skeleton.

[0092] Time-Scale Modification (TSM) is an audio processing technology that can change the playback speed of audio while maintaining the same pitch.

[0093] Pitch Synchronous Overlap and Add (PSOLA) is an audio processing algorithm used for pitch shifting. Unlike WSOLA, PSOLA's main goal is to change the pitch of audio while preserving its duration and timbre as much as possible. The core idea of ​​PSOLA is to adjust the fundamental frequency (pitch) and resynthesize the signal through periodic analysis of the audio signal, thereby achieving pitch change.

[0094] It is understood that in the specific implementation of this application, data related to the operation information of the participating objects are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0095] The design concept of the embodiments of this application will be introduced below.

[0096] With the rapid development of computer and internet technologies, various applications with audio-visual functions have emerged. Participants can use these applications to experience audio-visual entertainment anytime, anywhere on the internet, and interact with other participants in different geographical locations.

[0097] With the relevant technology, online chorus is an interactive entertainment method that allows participants from different geographical locations to sing a song together. During the chorus, each participant listens to the accompaniment played on their local terminal and sings along with the other participants; the local terminal collects the audio signal and sends it to a mixing server via the internet. The mixing server mixes the received multiple audio signals to obtain a mixed signal, which is then returned to each terminal for playback. During the performance, the application can display multi-dimensional scores of the current singing level in real time, and can also record the performance to generate a publishable work for a wider audience.

[0098] However, in actual choral singing, the singing quality of different participants may vary. This quality may be related to the participant's singing ability, the equipment's sound quality, or network latency. Therefore, the quality of the audio signals collected from different participants will also differ. In this situation, directly mixing the audio signals from different participants may result in some loud but low-quality audio signals becoming the core sound parts of the mixed signal, leading to a poor mixed signal quality and affecting the overall choral effect. Secondly, in actual choral singing, there is a lack of interaction between the participants, which also affects the choral effect.

[0099] In view of this, embodiments of this application provide an audio processing method, which includes the following steps:

[0100] In response to the first participating object, the chorus participation operation triggered by the command role in the target application is displayed, and the first interface of the chorus virtual room under the command role is displayed. The first interface includes: multiple singing object groups, and each singing object group includes: at least one second participating object that accesses the chorus virtual room as a singing role.

[0101] In response to the first participating object, an audio adjustment operation triggered for at least one singing object group is performed, and choral audio is played in the first interface; wherein, the choral audio is obtained by adjusting the original mixed audio of each of the at least one singing object group separately, and then mixing the obtained at least one target mixed audio with the original mixed audio of each of the other singing object groups.

[0102] In this embodiment of the application, the first participant triggers the chorus participation operation in the target application as a conductor, so that the target application displays the first interface of the chorus virtual room under the conductor's role. In this way, the first participant can participate in the chorus interaction in a brand-new role as a conductor, so that the participant can experience different interactive effects and auditory experience.

[0103] Secondly, in response to the first participating object, for the audio adjustment operation triggered by at least one singing object group, the original mixed audio of each of the at least one singing object group is adjusted separately, and the obtained target mixed audio is mixed with the original mixed audio of each of the other singing object groups to obtain the chorus audio. In this way, the first participating object can adjust the mixed audio of each singing object group according to the actual situation, effectively avoiding some audio signals with large volume but low quality from becoming the core sound part of the chorus audio, thereby effectively improving the quality of the chorus audio and thus improving the chorus effect.

[0104] In addition, the first participants can adjust the unique choral audio according to their own preferences, which can make the content of the choral audio richer and more interesting, while increasing the fun and participation of the participants in the choral interaction.

[0105] The following is a brief introduction to the system architecture diagram applicable to the technical solutions of the embodiments of this application. It should be noted that the system architecture diagram described below is only used to illustrate the embodiments of this application and is not intended to limit the scope of the application.

[0106] refer to Figure 1 This is a system architecture diagram applicable to the embodiments of this application. The system architecture includes at least terminal device 101 and server 102. The number of terminal devices 101 can be one or more, and the number of servers 102 can also be one or more. This application does not specifically limit the number of terminal devices 101 and servers 102.

[0107] Terminal device 101 can run target applications with audio processing capabilities. The types of target applications include: client applications, web applications, mini-program applications, etc. Terminal device 101 can be an Extended Reality (XR) device, smartphone, tablet, laptop, desktop computer, smart home appliance, smart voice interaction device, smart in-vehicle device, etc., but is not limited to these.

[0108] In this application, multiple participants engage in choral interaction. These multiple participants can correspond to multiple target applications, each installed on a terminal device. Each participant can assume multiple roles in the choral interaction, including conductor, singer, and audience member.

[0109] Server 102 is the backend server for the target application. Server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0110] The audio processing method in this application embodiment can be executed by the terminal device 101 or the server 102 alone, or by the terminal device 101 and the server 102 together.

[0111] Both server 102 and terminal device 101 may include one or more processors, memory, and interactive I / O interfaces. Furthermore, server 102 may be configured with a database for audio data, etc. The memory of server 102 and terminal device 101 may also store program instructions required for execution in the audio processing method provided in this application embodiment. These program instructions, when executed by the processor, can be used to implement the audio processing procedure provided in this application embodiment.

[0112] It should be noted that when the audio processing method provided in this application embodiment is executed by either server 102 or terminal device 101 alone, the system architecture of this application may also include only a single device, either server 102 or terminal device 101. Alternatively, server 102 and terminal device 101 may be considered as the same device. Of course, in practical applications, when the audio processing method provided in this application embodiment is executed by both server 102 and terminal device 101, server 102 and terminal device 101 may also be the same device. That is, server 102 and terminal device 101 may be different functional modules of the same device, or virtual devices virtualized from the same physical device.

[0113] In this embodiment, the terminal device 101 and the server 102 can communicate directly or indirectly through one or more networks. The network can be a wired network or a wireless network; for example, the wireless network can be a mobile cellular network or a Wireless-Fidelity (WIFI) network, or other possible networks. This embodiment does not limit the types of networks used.

[0114] The following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application are applicable. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0115] The solution provided in this application can be applied to various interactive choral scenarios, such as online concerts, karaoke group singing, remote music teaching, and short video recording. This solution can also be used as a foundational technology in various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0116] The following provides an illustrative example of the application scenarios to which the solution provided in this application is applicable.

[0117] Application Scenario 1: Online Concert Scenario.

[0118] The target application is set as a concert application. The terminal device responds to participant A by initiating a chorus participation operation in the concert application under the direction of the conductor. The first interface of the chorus virtual room under the conductor's role is displayed. See details below. Figure 2 The first interface includes two groups of singing objects: the singer group 201 and the audience group 202. The singer group 201 includes the lead singer who enters the chorus virtual room as a singing character; the audience group 202 includes multiple audience objects who enter the chorus virtual room as singing characters.

[0119] In response to the volume adjustment operation triggered by participant A on the first interface, the terminal device increases the singing volume of singer group 201, making the lead singer's singing voice the core sound part; at the same time, it lowers the volume of the mixed audio of audience group 202, making the singing voice of audience group 202 the harmony part.

[0120] The adjusted audio of singer group 201 and the adjusted mixed audio of audience group 202 are mixed to obtain chorus audio, which is then played on the first interface.

[0121] Online concerts allow singers to interact and perform simultaneously with thousands of online viewers in different locations. This flexible format is not limited by venue, time, or weather, and the number of participants can far exceed that of offline concerts, enhancing interactivity.

[0122] Application Scenario 2: Karaoke group singing scenario.

[0123] The target application is set to a karaoke app. The terminal device responds to participant B's (the person in charge) chorus participation action triggered by the karaoke app, displaying the first interface of the virtual chorus room under the direction of the person in charge. See [link / reference]. Figure 3 The first interface includes three groups of participants: Children's Group 301, Youth Group 302, and Adult Group 303. Children's Group 301 includes multiple participants aged 6-12 who have joined the virtual chorus room as singers. Youth Group 302 includes multiple participants aged 13-18 who have joined the virtual chorus room as singers. Adult Group 303 includes multiple participants aged 19 and above who have joined the virtual chorus room as singers.

[0124] In response to the audio adjustment operation triggered by participant B in the first interface, the terminal device increases the speech rate of the mixed audio in children's group 301 and increases the volume of the mixed audio in adult group 303.

[0125] The adjusted mixed audio of children's group 301, the adjusted mixed audio of adult group 303, and the unadjusted mixed audio of youth group 302 are then mixed to obtain the choral audio.

[0126] After obtaining the chorus audio, you can play it in real time on the first interface, save it, or share or forward it online.

[0127] It should be noted that the embodiments of this application are not limited to the application scenarios exemplified above, but may also be other application scenarios, which are not specifically limited in this application.

[0128] based on Figure 1 The system architecture diagram shown in this application illustrates the flow of an audio processing method. Figure 4 As shown, the process of this method is executed by a computer device, which can be... Figure 1 The terminal device 101 and / or server 102 shown include the following steps:

[0129] Step 401: In response to the first participating object, the chorus participation operation triggered by the command role in the target application is displayed, showing the first interface of the chorus virtual room under the command role.

[0130] Specifically, the target application can be an application with chorus functionality, such as a karaoke app or a concert app. A virtual chorus room refers to a virtual space that allows participants from different locations to sing together in real time via the internet. In the target application, the virtual chorus room is presented through a virtual room interface.

[0131] The target application offers multiple roles for participants to choose from and supports switching between roles. These roles include: conductor, singer, and audience member.

[0132] The conductor is the one who controls the effect of the chorus during the singing process; the singer is the one who participates in the chorus or sings solo; and the audience is the one who simply listens to the audio of the solo or chorus.

[0133] In the target application, each role corresponds to a unique functional module. For example, the conductor role corresponds to the functional module for controlling the chorus effect in the target application. Of course, each role also corresponds to some general functional modules; for example, the conductor role corresponds to the audio and video acquisition module in the target application.

[0134] Participation in a choral performance can involve creating and entering a virtual choral room, or directly entering an existing virtual choral room. This existing virtual room can be one created by the first participant or by other participants. Participation can be achieved through screen taps, swipes, voice input, and gesture input.

[0135] The virtual room interface for choral singing differs depending on the role. When the first participant enters the choral singing virtual room as the conductor, the target application displays the virtual room interface under the conductor role (i.e., the first interface). The first interface includes multiple groups of singing participants, and each group of singing participants includes at least one second participant who enters the choral singing virtual room as a singer.

[0136] Multiple singing groups are displayed in different areas of the first interface. For each singing group, the first interface displays the identification information of each second participating object within that group. This identification information can include: the second participating object's avatar (account avatar or character avatar), name (account name or character name), and ID number. Additionally, the identification information for each singing group can also be displayed, which can include: the group's name and ID number.

[0137] In some embodiments, in addition to including multiple groups of singing objects, the first interface may also include identification information of the first participating object. The identification information of the first participating object may be: the first participating object's avatar (account avatar or character avatar), name (account name or character name), number, etc.

[0138] For example, see Figure 5This is a schematic diagram of a first interface provided in an embodiment of this application. The first interface includes three singing groups: male voice group 501, female voice group 502, and children's voice group 503. Male voice group 501 includes the name of the singing group "Boys Group" and the avatars of the three participants; female voice group 502 includes the name of the singing group "Female Voice Group" and the avatars of the three participants; children's voice group 503 includes the name of the singing group "Children's Voice Group" and the avatars of the three participants.

[0139] Among them, male voice group 501 is displayed in the lower left area of ​​the first screen, female voice group 502 is displayed in the lower right area of ​​the first screen, and children's voice group 503 is displayed in the upper middle area of ​​the first screen.

[0140] The first interface also includes: identification information 504 of participant A, who has joined the chorus virtual room as a conductor. Participant A's identification information includes: participant A's character avatar and character name "Conductor". Participant A's identification information 504 is displayed in the lower middle area of ​​the first interface.

[0141] In some embodiments, in response to a chorus participation operation triggered by a first participating object and a director role in the target application, before displaying the first interface of the chorus virtual room under the director role, in response to a role selection operation triggered by the first participating object in the target application, the role of the first participating object is set to the director role.

[0142] Specifically, the role selection operation described above can be either the initial setting of a role or the switching of an already set role. The already set role can be the system default or a role previously set by the first participant. The following sections provide a detailed description of several role setting scenarios:

[0143] Upon initial launch of the target application, a role selection interface can be displayed, offering multiple selectable roles. The first participant triggers a role selection action in this interface, choosing the command role from the available options. In response to this action, the first participant's role is set as the command role. Subsequent actions by the first participant in the target application as the command role will display the corresponding interface for that role. Subsequent launches of the target application will directly reuse the previously set role.

[0144] Alternatively, each time the target application is launched, a role selection interface can be displayed in the target application, and then, in response to the role selection operation triggered by the first participating object in the role selection interface, the role of the first participating object can be set as the command role.

[0145] When the first participant triggers the chorus function module in the target application, a role selection interface can be displayed in the target application; then, in response to the role selection operation triggered by the first participant in the role selection interface, the role of the first participant can be set as the conductor role.

[0146] After setting the role of the first participating object to the conductor role using any of the above methods, in response to the first participating object, the first interface of the chorus virtual room under the conductor role is displayed based on the chorus participation operation triggered by the conductor role in the target application.

[0147] For example, see Figure 6 In response to the karaoke application launch operation triggered by participant A, the terminal device displays the application homepage 601 of the karaoke application, and displays the role selection interface 602 in the form of a pop-up window on the application homepage 601. The role selection interface 602 includes three selectable roles: conductor role, singer role, and audience role.

[0148] Participant A selects a command role in the role selection interface 602. The terminal device responds to Participant A's role selection operation, sets Participant A's role as the command role, and displays the selected command role in area 603 of the application homepage 601.

[0149] In this embodiment of the application, by providing a variety of roles for participants to choose from, each participant can select the corresponding role to participate in the choral interaction according to their preferences. This not only improves the experience of the participants, but also enhances the richness of the content of the target application.

[0150] In some embodiments, after the first participating object's role is set to the command role in the target application, the first participating object can switch the command role to another role at any subsequent time. Several role switching methods are illustrated below:

[0151] Method 1: In response to the role switching operation triggered by the first participant in the first interface, the role of the first participant is switched to the singing role, and the second interface of the chorus virtual room under the singing role is displayed.

[0152] Specifically, role switching can be achieved through screen taps, screen swipes, voice input, and gesture input. Taking one of these methods, the first interface displays the current role (i.e., the command role) of the first participant, along with a role switching button. Clicking the role switching button triggers a role selection interface on the first screen, which includes multiple selectable roles.

[0153] When the first participant selects a singing role in the role selection interface, in response to the role switching operation triggered by the first participant, the first participant's role is switched from the conductor role to the singing role, and the second interface of the chorus virtual room under the singing role is displayed.

[0154] The second interface displays a chorus group, which includes a first participant and multiple second participants who have joined the virtual chorus room as singers; alternatively, the second interface may display multiple groups of singers, with the first participant residing in one of these groups. The second interface can also take other forms, which are not specifically limited in this application.

[0155] It should be noted that the first participant can also trigger a role switching operation on the second interface to switch the first participant's role back to the conductor role and display the first interface of the chorus virtual room under the conductor role. This application does not make specific limitations on this.

[0156] For example, see Figure 7 The first interface 700 is displayed in the target application. The first interface 700 includes: male voice group 501, female voice group 502, children's voice group 503, identification information of participant A 504, and role switching button 701.

[0157] Participant A clicks the role switching button 701, triggering the display of the role selection interface 702 in the first interface 700. The role selection interface 702 includes: conductor role, singer role, and audience role, among which the conductor role is shown to have been selected.

[0158] Participant A selects a singing role in the role selection interface 702. The terminal device responds to Participant A's role selection operation, sets Participant A's role as the singing role, and switches from the first interface to the second interface of the chorus virtual room under the singing role.

[0159] The following are several ways to implement the second interface:

[0160] See Figure 8A The second interface 800 includes: new identification information 801 for participant A and chorus group 802. The new identification information 801 for participant A is located in chorus group 802. Chorus group 802 also includes: avatars of each participant in male voice group 501, female voice group 502, and children's voice group 503. The new identification information 801 includes: the character avatar and character name "Singer" of participant A.

[0161] See Figure 8BThe second interface 800 includes: the new identification information 801 of the switched participant A, the male voice group 501, the female voice group 502, and the children's voice group 503. Among them, the new identification information 801 is located in the female voice group 502. The new identification information 801 includes: the character avatar and character name "Singer" of participant A.

[0162] Method 2: In response to the role switching operation triggered by the first participant in the first interface, the role of the first participant is switched to the audience role, and the third interface of the chorus virtual room is displayed under the audience role.

[0163] Specifically, the first interface displays the current role (i.e., the command role) of the first participant, along with a role switching button. Clicking the role switching button triggers a role selection interface on the first screen, which includes multiple selectable roles.

[0164] When the first participant selects the audience role in the role selection interface, in response to the role switching operation triggered by the first participant, the first participant's role is switched from the conductor role to the audience role, and the third interface of the chorus virtual room under the audience role is displayed.

[0165] The third interface separately displays the current role (i.e., the audience role) and other identifying information of the first participant. Additionally, the third interface displays a chorus group, which includes multiple second participants who have joined the virtual chorus room as singers; alternatively, multiple second participants can be displayed as a group of singers on the third interface. Of course, the third interface can also take other forms, and this application does not specifically limit its application to these forms.

[0166] It should be noted that the first participant can also trigger a role-switching operation on the third interface, switching their role back to the conductor role and displaying the first interface of the chorus virtual room under the conductor role. Of course, the first participant can also trigger a role-switching operation on the third interface, switching their role to the singer role and displaying the second interface of the chorus virtual room under the singer role; this application does not specifically limit this.

[0167] For example, see Figure 9 The first interface 700 is displayed in the target application. The first interface 700 includes: male voice group 501, female voice group 502, children's voice group 503, identification information of participant A 504, and role switching button 701.

[0168] Participant A clicks the role switch button 701, triggering the display of the role selection interface 702 on the first interface 700. Participant A selects the audience role on the role selection interface 702. The terminal device responds to Participant A's role selection operation, sets Participant A's role to the audience role, and switches from the first interface 700 to the third interface of the chorus virtual room under the audience role.

[0169] The following are several ways to implement a third interface:

[0170] See Figure 10A The third interface 1000 includes: new identification information 1001 for participant A and chorus group 1002. The chorus group 1002 includes: the avatars of each participant in the male voice group 501, female voice group 502, and children's voice group 503. The new identification information 1001 includes: the character avatar and character name "audience" of participant A.

[0171] See Figure 10B The third interface 1000 includes: the new identification information 1001 of the switched participant A, the male voice group 501, the female voice group 502, and the children's voice group 503. Among them, the new identification information 1001 includes: the character avatar and character name "audience" of participant A.

[0172] In this embodiment of the application, the target application provides a function for switching between multiple roles. Therefore, each participant can switch roles at any time during the chorus to obtain a richer chorus experience brought by different roles, thereby increasing the fun of chorus.

[0173] It should be noted that, in this application, in addition to supporting role switching after the first participant enters the virtual chorus room, role switching can also be performed on any interface before entering the virtual chorus room or on any interface after exiting the virtual chorus room. This application does not make any specific limitations on this.

[0174] In some embodiments, the multiple singing groups displayed in the first interface are divided according to a preset first grouping mode. To ensure that when the first interface is displayed, it correspondingly displays multiple singing groups divided according to the preset first grouping mode, in this embodiment, in response to the chorus participation operation triggered by the first participating object and the command role in the target application, before displaying the chorus virtual room on the first interface under the command role, in response to the mode selection operation triggered by the first participating object in the target application, the singing grouping mode is set to the first grouping mode.

[0175] Specifically, the target application provides multiple singing grouping modes for participants to choose from, and supports switching between these modes on any interface. The singing grouping modes provided by the target application include: vocal range grouping mode, age grouping mode, and voice part grouping mode, etc., and the grouping characteristics are defined differently under each mode. The first grouping mode can be any of the multiple singing grouping modes provided by the target application.

[0176] In practical applications, the above mode selection operation can be either setting the grouping mode for the first time or switching an existing grouping mode. The existing grouping mode can be the system default or a mode previously set by the first participating object. The following sections provide a detailed introduction to several grouping mode setting scenarios:

[0177] This can be achieved by displaying multiple selectable singing group modes within the target application upon initial launch. The first user triggers a mode selection operation within the target application, choosing one singing group mode (i.e., the first group mode) from the multiple options. The terminal device responds to the mode selection operation by setting the target application's singing group mode to the first group mode. Subsequent launches of the target application can directly reuse the previously set singing group mode.

[0178] It should be noted that if you need to set the grouping mode and the roles of the participants when you first launch the target application, you can display multiple roles and multiple singing grouping modes on the same screen. In this way, the participants can complete the role setting and grouping mode setting in one screen. Alternatively, you can display the role selection screen containing multiple roles and the grouping setting screen containing multiple singing grouping modes in a preset order, and then complete the role setting and grouping mode setting in sequence.

[0179] In addition, this application can also display a grouping settings interface in the target application when the role of the first participating object is set to the command role. The grouping settings interface includes: multiple selectable singing grouping modes; and then, in response to the mode selection operation triggered by the first participating object, the singing grouping mode is set to the first grouping mode.

[0180] In addition, this application can also display a group setting interface in the target application when the first participating object triggers the chorus function module in the target application; then, in response to the mode selection operation triggered by the first participating object in the group setting interface, the singing group mode is set to the first group mode.

[0181] After setting the singing grouping mode to the first grouping mode using any of the above methods, in response to the chorus participation operation triggered by the first participating object and the command role in the target application, the first interface of the chorus virtual room under the command role is displayed. At this time, the first interface includes: multiple singing object groups divided according to the first grouping mode.

[0182] For example, see Figure 11 , set according to Figure 6 After setting participant A's role as the conductor as shown, the group settings interface 1101 is displayed as a pop-up window on the application homepage 600. The group settings interface 1101 includes three singing grouping modes: vocal range grouping mode, age grouping mode, and voice part grouping mode.

[0183] Participant A selects the vocal range grouping mode in the grouping settings interface 1101. The terminal device responds to Participant A's mode selection operation by setting the singing grouping mode of the karaoke application to the vocal range grouping mode and displays the selected vocal range grouping mode in area 1102 of the application homepage 600.

[0184] In this embodiment of the application, multiple singing grouping modes are provided in the target application for participants to choose from. Participants can select the corresponding singing grouping mode according to their preferences. In this way, when a participant acts as a conductor to control the choral effect, they can divide the participants into multiple singing groups according to the selected singing grouping mode and control the audio of multiple singing groups to mix, thereby obtaining a better choral effect.

[0185] In some embodiments, the multiple singing object groups in the first grouping mode are obtained using the following division method:

[0186] For each second participant who enters the chorus virtual room as a singer, the following steps are performed: extracting the voice features of the second participant from its historical singing data; and obtaining the group category corresponding to the second participant in the first grouping mode based on the voice features of the second participant.

[0187] Specifically, the historical performance data of the second participant includes: audio data generated during the second participant's past solo performances and / or audio data generated during past choral performances. In some cases, the historical performance data of the second participant may also include: basic information about the second participant and performance score data, etc.

[0188] In the embodiments of this application, a classification model can be used to predict the group category corresponding to the second participating object in the first grouping mode based on the sound features of the second participating object; alternatively, the sound features of the second participating object can be compared with the threshold ranges of multiple group categories in the first grouping mode to obtain the group category corresponding to the second parameter; other implementation methods can also be used to obtain the group category corresponding to the second parameter.

[0189] The following provides a detailed explanation of several of these implementation methods:

[0190] Implementation Method 1: Using a classification model, based on the voice characteristics of the second participant, predict the group category corresponding to the second participant in the first grouping mode.

[0191] Specifically, a corresponding classification model is pre-trained for the first grouping pattern. The classification result of this model includes multiple grouping categories under the first grouping pattern. The model structure used by the classification model can be any model structure capable of classification, such as Convolutional Neural Network (CNN), LSTM model, Recurrent Neural Network (RNN) model, transformer model, encoder-decoder architecture model, or GPT series models, etc.

[0192] In this embodiment of the application, the training process of the classification model specifically includes the following steps:

[0193] An audio sample set is constructed, containing multiple audio samples. Each audio sample includes: historical singing data of a participant and its corresponding true group category. Using this audio sample set, the classification model to be trained is iteratively trained multiple times until the training termination condition is met, thus obtaining the trained classification model. Each round of iterative training includes the following steps:

[0194] The classification model used in this round identifies each audio sample input in this iteration process and obtains the predicted grouping category of each audio sample. Then, based on the obtained predicted grouping category and the corresponding real grouping category, the model loss value of the classification model used in this round is determined. After tuning the parameters of the classification model used in this round based on the model loss value, the tuned classification model is used to enter the next round of iteration training.

[0195] By continuously optimizing the model parameters, the model can meet the accuracy requirements. When this iteration is the first iteration, the model used in this iteration is the initial model; when this iteration is not the first iteration, the model used in this iteration is the model tuned in the previous iteration. During each iteration of training, some or all audio samples can be extracted from the audio sample set and input into the model used in this iteration. For example, random selection can be used, or the audio sample set can be pre-divided into batches, with one batch of audio samples input each time. The training is then based on these input audio samples, meaning that the model used in this iteration can perform forward inference on each audio sample.

[0196] In this embodiment, the model loss value can also be the cross-entropy loss function (CE loss), the mean squared error (MSE) loss function, the squared absolute error loss function, the maximum likelihood loss (LHL) function, etc. Of course, it can also be other possible loss functions, and this embodiment does not limit them.

[0197] In this embodiment of the application, the iteration termination condition may include at least one of the following conditions:

[0198] (1) The number of iterations reaches the set threshold.

[0199] (2) The model loss value is less than the set loss threshold, or the model loss value reaches the minimum value.

[0200] After training and obtaining the classification model for the first grouping pattern, the historical singing data of the second participant is acquired. The historical singing data of the second participant includes: audio data generated when the second participant sang solo in the past and / or audio data generated when participating in chorus in the past.

[0201] In addition, the vocal characteristics of the second participant can be extracted by combining the historical singing data of the second participant and the real-time audio data generated by the second participant in the virtual choral room. The vocal characteristics of the second participant include at least one of the following: MFCC, fundamental frequency characteristics, and formant characteristics.

[0202] Specifically, MFCC can be obtained as follows: The audio data is segmented and windowed to obtain multi-frame audio signals. A Fast Fourier Transform (FFT) is performed on each frame to obtain the spectrum. The linear frequency scale of the spectrum is converted to a Mel scale. A set of triangular filters (typically 20-40) is designed and uniformly distributed on the Mel scale. The spectrum is weighted and summed to obtain the energy of each triangular filter. The logarithm and discrete cosine transform of the energy of the triangular filter bank are then applied to obtain the MFCC.

[0203] Specifically, the fundamental frequency characteristics can be obtained in the following ways: First, calculate the autocorrelation function of the audio data and find the period corresponding to the maximum peak value. Then, the fundamental frequency = 1 / period. Then, find the fundamental frequency and its harmonics through spectrum analysis. Finally, use short-time Fourier transform (STFT) or wavelet transform to extract the fundamental frequency characteristics.

[0204] Specifically, formant features can be obtained in the following ways: First, use a linear predictive coding model to estimate the spectral envelope of the audio data and extract the formant frequencies from the spectral envelope; then, separate the vocal tract response and excitation signal through cepstral analysis and extract the formant features from the vocal tract response; finally, perform peak detection on the spectrum to find the frequencies with concentrated energy as formant features.

[0205] After obtaining the vocal features of the second participant using the above method, the vocal features of the second participant are input into the classification model corresponding to the first grouping mode for prediction to obtain the grouping category of the second participant in the first grouping mode. Then, based on the grouping categories of the multiple second participants connected to the chorus virtual room, the multiple second participants are divided into multiple singing object groups in the first grouping mode.

[0206] For example, when the first grouping mode is the vocal range grouping mode, the multiple grouping categories under the vocal range grouping mode include: male voice group, female voice group, and children's voice group. A vocal range classification model is pre-trained for the vocal range grouping mode. The training process of the classification model has been introduced earlier and will not be repeated here.

[0207] For each second participant, historical and real-time audio data are collected, and vocal features are extracted from these data. These features are then input into a voice classification model for prediction, yielding a classification result for the second participant, which can be any one of three groups: male, female, or children's voices. The classification results for multiple second participants are then used to categorize them into these groups.

[0208] For example, when the first grouping pattern is the age grouping pattern, the multiple grouping categories under the age grouping pattern include: children (aged 6-12), teenagers (aged 13-18), and adults (aged 19 and above). An age classification model is pre-trained for the age grouping pattern.

[0209] For each second participant, historical and real-time audio data are collected, and voice features are extracted from these data. These voice features are then input into an age classification model for prediction, yielding a classification result for the second participant, which can be any one of three groups: children, adolescents, or adults. The classification results for multiple second participants are then combined to categorize them into these groups.

[0210] Implementation Method 2: The sound features of the second participating object are compared with the threshold ranges of multiple grouping categories in the first grouping mode to obtain the grouping category corresponding to the second parameter.

[0211] Specifically, the vocal features of the second participant are extracted from their historical singing data; these vocal features are expressed as numerical values. For example, the age of the second participant can be extracted from their basic information; or, their overall singing score can be extracted from their singing score data. The vocal features are then compared with threshold ranges for multiple group categories, and the group category corresponding to the threshold range in which the vocal features fall is taken as the group category for the second participant.

[0212] For example, when the first grouping mode is the age grouping mode, the multiple grouping categories under the age grouping mode include: children (corresponding age threshold range: 6-12 years old), adolescents (corresponding age threshold range: 13-18 years old) and adults (corresponding age threshold range: 19 years old and above).

[0213] For each second participant, their age is extracted from their basic information. This age is then compared to the age threshold ranges for each of the multiple grouping categories to determine the participant's grouping category: children, adolescents, or adults. Based on these grouping categories, the participants are then divided into children, adolescents, and adults.

[0214] For example, when the first grouping mode is the vocal grouping mode, the multiple group categories under this mode include: Lead Singer Group (corresponding scoring threshold range: 85 points and above), Core Group (corresponding scoring threshold range: 70-85 points), and Trainee Group (corresponding scoring threshold range: below 70 points). The vocal grouping mode is closely related to the participants' singing experience level. The Lead Singer Group has better experience and skill, and more precise control over pitch and rhythm. The Core Group has average experience and skill, and relatively accurate control over pitch and rhythm. The Trainee Group has weaker experience and skill, and poorer control over pitch and rhythm.

[0215] For each second participant, multiple singing scores are extracted from their singing score data, and each score is assigned a corresponding weight. Among the multiple singing scores, the further back in time the score was generated, the lower its weight; the more recent the score was generated, the higher its weight. Using the respective weights of the multiple singing scores, a weighted sum is calculated to obtain the comprehensive singing score of the second participant.

[0216] The overall singing score of the second participant is compared with the score threshold ranges of each of the above-mentioned group categories to obtain the group category of the second participant, which can be any one of the following: lead singer group, core group, or student group. Based on the group categories of the multiple second participants, they are divided into the lead singer group, core group, and student group.

[0217] In this embodiment, the participants are divided into multiple singing groups based on their individual vocal characteristics. Then, the mixed audio of the multiple singing groups is adjusted and processed according to the instructions of the conductor to obtain the final chorus audio, thereby improving the quality of the chorus audio and enhancing the interactive effect and auditory experience of the chorus.

[0218] In some embodiments, when displaying the first interface of the virtual chorus room under the conductor's role, the first interface includes: multiple groups of singing participants according to a first grouping mode. The first participating participant can switch the first grouping mode to another grouping mode at any subsequent time.

[0219] Specifically, in response to the mode switching operation triggered by the first participating object in the target application, the singing group mode is switched to the second group mode, and multiple singing object groups in the second group mode are displayed in the first interface.

[0220] In this embodiment of the application, the second grouping mode can be any one of the multiple singing grouping modes provided by the target application, and the second grouping mode is different from the first grouping mode.

[0221] In practice, mode switching can be achieved through tapping the screen, swiping the screen, voice input, gesture input, etc. Taking one of these methods, a mode switching button is displayed on the first interface. When the first participant taps the mode switching button, it triggers the display of the group settings interface on the first screen, which includes multiple singing group modes.

[0222] When the first participating object selects the second group mode in the group settings interface, in response to the mode switching operation triggered by the first participating object, the singing group mode of the target application is switched from the first group mode to the second group mode, and multiple singing object groups in the second group mode are displayed in the first interface.

[0223] It should be noted that the first participant can also trigger a mode switching operation on the first interface to switch the target application's singing group mode from the second group mode back to the first group mode, or switch from the second group mode to other singing group modes besides the first group mode. This application does not make specific limitations on this.

[0224] For example, see Figure 12 The first interface 700 is displayed in the target application. The first interface 700 includes: male voice group 501, female voice group 502, children's voice group 503, identification information of participant A 504, and mode switching button 1201.

[0225] When participant A clicks the mode switching button 1201, it triggers the display of the group setting interface 1202 on the first interface 700. The group setting interface 1202 includes: vocal range grouping mode, age grouping mode, and voice grouping mode. Among them, the vocal range grouping mode is shown to have been selected.

[0226] Participant A selects the vocal grouping mode in the grouping settings interface 1202. The terminal device responds to Participant A's mode switching operation and displays the lead singer group 1203, the backbone group 1204, and the trainee group 1205 in the first interface 700.

[0227] In this embodiment of the application, multiple singing group modes are provided in the target application for participants to choose from. In this way, when participants act as conductors to control the choral effect, they can switch between singing group modes according to their preferences in order to experience the choral interaction under different group modes, thereby obtaining a better choral effect and interactive experience.

[0228] Step 402: In response to the first participating object, an audio adjustment operation triggered for at least one singing object group is performed, and the chorus audio is played in the first interface.

[0229] Specifically, for each second participant, after the second participant enters the chorus virtual room as a singer in the corresponding target application, the terminal device corresponding to the second participant collects the real-time audio data of the second participant; then the collected real-time audio data is encoded and compressed to obtain the corresponding compressed audio data, and the compressed audio data is sent to the server.

[0230] The server can forward the compressed audio data of each second participant to the terminal device corresponding to the first participant. The terminal device can then decode the compressed audio data of each second participant to obtain the corresponding real-time audio data. Then, for each singing group, the real-time audio data of each second participant within the singing group can be mixed to obtain the corresponding original mixed audio and save it.

[0231] In practical applications, the real-time audio data of multiple second participating objects can be weighted and summed in the time domain according to their respective mixing weights to obtain the original mixed audio. The mixing weight of the real-time audio data of each second participating object can be obtained as follows: obtain the maximum sample value of the real-time audio data of each second participating object, and obtain the maximum sample value of the accumulated audio of the real-time audio data of multiple second participating objects. For each second participating object, the ratio of the maximum sample value of the real-time audio data of the second participating object to the maximum sample value of the accumulated audio is used as the mixing weight of the real-time audio data of that second participating object.

[0232] When the first participating entity triggers an audio adjustment operation for at least one singing group in the target application, the terminal device adjusts the original mixed audio of each of the at least one singing group separately, and then uses the obtained at least one target mixed audio to mix with the original mixed audio of each of the other singing groups to obtain a chorus audio, which is then played. The mixing process has been described above and will not be repeated here.

[0233] In addition, the terminal device can also synchronize the role information and singing grouping mode set by the first participant to the server in real time. This allows the server to divide multiple second participants into multiple singing group groups according to the singing grouping mode. After receiving the compressed audio data from each of the multiple second participants, the server decodes the compressed audio data to obtain and save the corresponding real-time audio data. Subsequently, for each singing group, the real-time audio data of each second participant within that group is mixed to obtain the corresponding original mixed audio.

[0234] When the first participating entity triggers an audio adjustment operation for at least one singing group in the target application, the terminal device sends a corresponding audio adjustment command to the server. After the server adjusts the original mixed audio of each of the at least one singing group, it uses the obtained target mixed audio to mix with the original mixed audio of other singing groups to obtain a chorus audio; and sends the chorus audio to the terminal device of the first participating entity for playback.

[0235] The aforementioned choral audio is a mixed audio obtained under the direction of the first participant. In addition to playing the choral audio in real time on the terminal device of the first participant, the choral audio can also be saved or shared and displayed online to gain likes and attention.

[0236] In addition to playing the choral audio obtained by the conductor on the terminal device of the first participant, the choral audio obtained by the conductor can also be played on the terminal device of each second participant; of course, the mixed audio without a conductor (i.e., the mixed audio obtained by mixing the real-time audio data of multiple second participants according to a fixed mixing algorithm) can also be played on the terminal device of each second participant; or, depending on the selection operation of each second participant, the choral audio obtained by the conductor or the mixed audio without a conductor can be played on the corresponding terminal device. This application does not make specific limitations on this.

[0237] In this embodiment of the application, the first participant triggers the chorus participation operation in the target application as a conductor, so that the target application displays the first interface of the chorus virtual room under the conductor's role. In this way, the first participant can participate in the chorus interaction in a brand-new role as a conductor, so that the participant can experience different interactive effects and auditory experience.

[0238] Secondly, in response to the first participating object, for the audio adjustment operation triggered by at least one singing object group, the original mixed audio of each of the at least one singing object group is adjusted separately, and the obtained target mixed audio is mixed with the original mixed audio of each of the other singing object groups to obtain the chorus audio. In this way, the first participating object can adjust the mixed audio of each singing object group according to the actual situation, effectively avoiding some audio signals with large volume but low quality from becoming the core sound part of the chorus audio, thereby effectively improving the quality of the chorus audio and thus improving the chorus effect.

[0239] In addition, the first participants can adjust the unique choral audio according to their own preferences, which can make the content of the choral audio richer and more interesting, while increasing the fun and participation of the participants in the choral interaction.

[0240] In some embodiments of this application, the chorus audio is obtained at least in the following ways:

[0241] The process involves acquiring the body movements of the first participating object; then, based on the body movements, obtaining audio adjustment instructions for at least one singing object group; adjusting the original mixed audio for each singing object group according to the corresponding audio adjustment instructions to obtain the corresponding target mixed audio; finally, using the obtained target mixed audio, mixing it with the original mixed audio of other singing object groups to obtain the chorus audio.

[0242] Specifically, in response to an audio adjustment operation triggered by the first participating object for at least one group of singing objects, a video segment of the first participating object is captured by an image acquisition device (such as a camera); then the captured video segment is identified to obtain the body movements of the first participating object;

[0243] In some embodiments, audio adjustment operations can be performed via an external device (such as a wireless remote control, smartwatch, etc.). The motion sensor of the external device can detect and analyze the limb movements of the first participant in real time.

[0244] Based on the body movements of the first participant, the system queries pre-defined mapping data to obtain audio adjustment instructions for at least one group of singers. These audio adjustment instructions include: volume adjustment, speech rate adjustment, pitch adjustment, start singing, and stop singing.

[0245] Volume adjustment commands include volume up and volume down commands, which are implemented through signal gain operations. For example, gain processing is applied to the audio signal sampling points in the original mixed audio of the singing group to obtain the target mixed audio of the singing group.

[0246] The speech rate adjustment commands include speech rate increase and speech rate decrease commands, which are implemented through waveform similarity overlay operations. For example, for the original mixed audio of a group of singers, the original mixed audio is first divided into frames of fixed length, resulting in multiple audio frames with overlap between adjacent frames. For each audio frame, the similarity between the audio frame and its neighboring audio frames is calculated to obtain the most similar matching frame. This matching frame is then overlaid and added to obtain the target mixed audio of the group of singers.

[0247] Pitch adjustment commands include pitch increase and pitch decrease commands, implemented through fundamental frequency synchronization overlay. For example, for the original mixed audio of a group of singers, the original mixed audio is first divided into frames of fixed length, resulting in multiple audio frames with overlap between adjacent frames. The fundamental frequency of each audio frame is estimated using an autocorrelation function or time-frequency analysis. Based on the target pitch, the fundamental frequency of each audio frame is adjusted. For example, if the pitch needs to be increased, the fundamental frequency period is shortened; if the pitch needs to be decreased, the fundamental frequency period is lengthened. The adjusted audio frames are then re-overlayed to obtain the target mixed audio of the group of singers.

[0248] According to the audio adjustment instructions corresponding to each singing group, the original mixed audio corresponding to the singing group is adjusted to obtain the corresponding target mixed audio; then, at least one target mixed audio is used to mix with the original mixed audio of each other singing group to obtain the chorus audio.

[0249] In this embodiment, the local device identifies the body movements of the participants and converts the body movements into preset audio adjustment instructions for the target application, so as to adjust the audio signals of each singing group, thereby producing a different auditory experience and enhancing the interactive experience of chorus.

[0250] In some embodiments, body movements include hand gestures. In this case, this application acquires the body movements of the first participant at least in the following ways:

[0251] Acquire a video clip of the first participating object, which consists of multiple frame images. Identify at least one set of joint points for each hand from each frame image.

[0252] Specifically, after obtaining multiple frame images, preprocessing operations are performed to remove interference factors in each frame image. These preprocessing operations include: Gaussian filtering for noise reduction, histogram equalization to enhance contrast, color gamut conversion, and adaptive threshold binarization.

[0253] A hand pose estimation model identifies at least one set of joints for each hand in each frame image, with each set containing multiple hand joints. Based on the set of joints for a hand, the hand skeleton is obtained, and the hand type can be further identified, including left and right hands.

[0254] For example, see Figure 13 The system detects 21 hand joints (including finger joints and palm), and generates a hand skeleton based on these 21 joints. Based on hand joints 0, 1, and 4, the natural bending direction of the thumb is obtained, and the hand type is determined to be "right-handed".

[0255] Next, for each hand, the following steps are performed: based on the obtained set of multiple joints of a hand, the motion features of that hand are extracted; and then based on the motion features of that hand, the hand gesture is obtained.

[0256] Specifically, for each set of joints, the coordinates of multiple hand joints within the set are normalized to form a feature vector. The feature vectors of each set of joints are then input into the gesture recognition model for identification. Within the gesture recognition model, the motion features of the hand are extracted based on the feature vectors of each set of joints; then, based on these motion features, the output probabilities for various predefined actions are obtained.

[0257] The hand gesture can be selected from predefined actions with the highest output probability. Alternatively, the output probabilities of multiple predefined actions can be compared with a preset threshold, and the predefined actions with probabilities greater than the preset threshold can be selected as the hand gesture. It should be noted that if the output probabilities of multiple predefined actions are all less than the preset threshold, it means that no clear hand gesture has been recognized, and therefore no subsequent instructions will be triggered.

[0258] The gesture recognition model used in this application can be any model structure capable of serialization processing, such as LSTM model, Recurrent Neural Network (RNN) model, transformer model, encoder-decoder architecture model, or GPT series model, etc.

[0259] In this embodiment, image sequences of participating objects are acquired by a local device, and then the gestures of participating objects are identified by capturing the long-term dependencies between the image sequences, thereby improving the accuracy of gesture recognition.

[0260] In some embodiments, this application uses at least the following methods to obtain audio adjustment instructions for the singing group:

[0261] For at least one frame image, the following steps are performed: Based on the set of joints of the hand in a frame image and the torso region identified from that frame image, the positional relationship between the hand and the torso region is obtained. Based on the obtained positional relationship, at least one singing object group controlled by the hand is determined; then, based on the hand's gesture, preset gesture mapping data is queried to obtain audio adjustment instructions for each of the at least one singing object group.

[0262] Specifically, the aforementioned at least one frame image can be all frame images in the captured video segment; or it can be a subset of frame images in the captured video segment. For example, a subset of frame images extracted from the video segment at preset frame intervals.

[0263] The human pose estimation model identifies key points of the torso, such as shoulders and hips, from frame images. Then, based on the obtained key points of the torso, the torso region in the frame image is obtained.

[0264] Because the hands of the first participant may move during choral conducting, the positional relationship between the hands and the torso will change accordingly. In practice, the positional relationship between the hands and the torso includes: the hands being inside the torso area and the hands being outside the torso area.

[0265] In practical applications, multiple display areas are predefined in the first interface, each area is used to display a group of singing objects; then the display areas are bound to positional relationships (i.e., the positional relationship between the hands and the torso area), where a positional relationship can be bound to one or more display areas.

[0266] Thus, during the chorus, after detecting the positional relationship between the first participant's hand and torso area, it is determined whether this positional relationship is bound to at least one display area. If so, the group of singing objects displayed in the bound display area is taken as the group of singing objects controlled by the first participant's hand. Of course, if the positional relationship between the hand and torso area changes during the chorus, the group of singing objects controlled by the first participant's hand is re-determined in the same way.

[0267] For example, the first interface has two display areas: an upper area and a lower area. The positional relationship "hands are within the torso area" is bound to the upper area of ​​the first interface, and the positional relationship "hands are outside the torso area" is bound to the lower area of ​​the first interface.

[0268] During the chorus, see Figure 2 The upper area of ​​the first interface displays the singer group 201, and the lower area displays the audience group 202. When it is detected that the hand of participant A is within the torso area, the audio of the singer group 201 is adjusted; when it is detected that the hand of participant A is outside the torso area, the audio of the audience group 202 is adjusted.

[0269] In some embodiments, at least one group of singing objects controlled by the hand is determined by combining the hand type and at least one obtained positional relationship.

[0270] Specifically, hand types include left hand and right hand. The display area, hand type, and positional relationship (i.e., the positional relationship between the hand and the torso area) are bound together. A hand type can be bound to one or more display areas, and a positional relationship can also be bound to one or more display areas. During choral singing, after detecting the positional relationship between the first participant's hand and torso area and their hand type, it is determined whether this positional relationship and hand type are bound to a display area. If so, the group of singing objects displayed in the bound display area is taken as the group of singing objects controlled by the first participant's hand. Of course, during choral singing, if the positional relationship between the hand and torso area changes, the same method is used to re-determine the group of singing objects controlled by the first participant's hand.

[0271] For example, the first interface is set with three display areas: the left area, the right area, and the middle area.

[0272] Bind the positional relationship "left hand outside the torso area" to the left area of ​​the first interface, bind the positional relationship "right hand outside the torso area" to the right area of ​​the first interface, and bind the positional relationship "left hand or right hand inside the torso area" to the middle area of ​​the first interface.

[0273] During the chorus, see Figure 5 The male voice group 501 is displayed in the left area of ​​the first interface, the female voice group 502 is displayed in the right area of ​​the first interface, and the children's voice group 503 is displayed in the middle area of ​​the first interface.

[0274] When it is detected that participant A's left hand is outside the torso area and his right hand is outside the torso area, audio adjustments are made to male voice group 501 and female voice group 502.

[0275] When it is detected that participant A's left hand is inside the torso area and his right hand is outside the torso area, audio adjustments are made to children's voice group 503 and female voice group 502.

[0276] When it is detected that participant A's left hand is outside the torso area and his right hand is inside the torso area, audio adjustments are made to the children's voice group 503 and the female voice group 502.

[0277] When it is detected that the left hand and right hand of participant A are both within the torso area, audio adjustment is performed on the children's voice group 503.

[0278] Next, based on the hand gestures, the system queries preset gesture mapping data to obtain audio adjustment instructions for at least one group of singing subjects. Specifically, the gesture mapping data includes audio adjustment instructions for multiple predefined gestures.

[0279] For example, see Figure 14 The gesture "spread fingers" corresponds to the audio adjustment command: volume up; the gesture "close fingers" corresponds to the audio adjustment command: volume down; the gesture "slide palm down" corresponds to the audio adjustment command: pitch down; and the gesture "slide palm up" corresponds to the audio adjustment command: pitch up.

[0280] In this embodiment, the local device identifies the gestures of the participants and converts them into preset audio adjustment commands for the target application. At the same time, based on the positional relationship between the hands and the torso, the controlled singing group is determined, thereby achieving accurate control of the audio signals of each singing group and improving the quality of the choral audio.

[0281] In some embodiments, body movements include lip movements. This application embodiment obtains the body movements of the first participant at least in the following ways:

[0282] A video clip of the first participant is acquired, consisting of multiple frames. The lip region is segmented from each frame; then, based on the segmented lip regions, lip movement features are extracted; and finally, based on the lip movement features, the mouth shape is obtained.

[0283] Specifically, the lip region in each frame image is located using a face detection model, such as Dib or MediaPipe.

[0284] The lip region image is segmented from each frame image, and each lip region image is scaled to a fixed size (e.g., 64*64 pixels). The obtained images of multiple lip regions are input into a lip shape recognition model, which can be a CNN+LSTM structure. In the lip shape recognition model, the CNN extracts features from each of the input lip region images, obtaining high-order features for each frame. The LSTM then performs sequence encoding on these high-order features to capture the temporal dependencies between them, obtaining lip movement features. Based on these lip movement features, the output probabilities for various predefined lip shapes are obtained.

[0285] The predefined lip shape with the highest output probability can be used as the recognized lip action. Alternatively, the output probabilities of multiple predefined lip shapes can be compared with a preset threshold, and the predefined lip shape with a probability greater than the preset threshold can be used as the recognized lip action. It should be noted that when the output probabilities of multiple predefined lip shapes are all less than the preset threshold, it means that no clear lip action has been recognized, and therefore no subsequent instructions will be triggered.

[0286] In this embodiment, the lip movements of the participants are identified by the local device and converted into audio adjustment instructions preset by the target application to adjust the audio signals of each singing group, thereby improving the quality of the choral audio and enhancing the auditory experience of the participants.

[0287] In some embodiments, this application uses at least the following methods to obtain audio adjustment instructions for the singing group:

[0288] Based on lip movements, query preset lip mapping data to obtain audio adjustment instructions for at least one singing target group.

[0289] Specifically, lip-sync actions can be used to simultaneously control multiple groups of singing objects displayed on the first interface; of course, this application can also pre-bind lip-sync actions to one or more display areas defined in the first interface. Thus, during chorus singing, when any lip-sync action is detected, the group of singing objects displayed in the bound display area will be used as the controlled group of singing objects. The lip-sync mapping data includes: audio adjustment instructions for each of the multiple predefined lip-sync actions.

[0290] For example, see Figure 15 The lip-sync gesture "from closed to open" corresponds to the audio adjustment command: volume up. The wider the mouth is opened, the greater the volume amplification for multiple singing groups. Conversely, the lip-sync gesture "from open to closed" corresponds to the audio adjustment command: volume down. The more the mouth is closed, the greater the volume reduction for multiple singing groups.

[0291] In this embodiment, the lip movements of the participants are identified by the local device, and the lip movements are quickly converted into preset audio adjustment commands of the target application by querying lip mapping data. The audio adjustment commands are then used to control the chorus, thereby improving the interactive effect of the chorus.

[0292] In some embodiments, body movements include: hand gestures and lip movements.

[0293] In practice, video clips of the first participating object are collected, and the video clips include multiple frame images. Based on the multiple frame images, the hand type, gesture, and positional relationship between the hand and the torso area are identified. Based on the hand type and at least one obtained positional relationship, at least one singing object group controlled by the hand is determined. Based on the hand gesture, preset gesture mapping data is queried to obtain audio adjustment instructions for each of the at least one singing object group. Similarly, based on the multiple frame images, the lip movements of the first participating object are identified. Based on the lip movements, preset lip-sync mapping data is queried to obtain audio adjustment instructions for each of the multiple singing object groups on the first interface. The specific recognition process has been described above and will not be repeated here.

[0294] According to the audio adjustment instructions corresponding to each singing group, the original mixed audio of the singing group is adjusted to obtain the corresponding target mixed audio. Then, at least one of the obtained target mixed audios is mixed with the original mixed audios of other singing groups to obtain the chorus audio.

[0295] In practical applications, a single vocal group may correspond to multiple audio adjustment commands. For example, if the multiple audio adjustment commands are: audio adjustment commands corresponding to gesture actions and audio adjustment commands corresponding to lip movements, then these multiple audio adjustment commands will be executed sequentially on the vocal group to obtain the corresponding target mixed audio.

[0296] Furthermore, multiple audio adjustment commands corresponding to a single singing target group can be of the same type or different types. For example, multiple audio adjustment commands can all be volume adjustment commands, or one of the multiple audio adjustment commands can be a volume adjustment command and another a pitch adjustment command.

[0297] In this embodiment, the first participant takes on a completely new role as a conductor in the choral interaction, allowing the first participant to experience a different interactive effect. Secondly, the local device recognizes the participant's body movements (gestures and lip movements) and converts the body movements into preset audio adjustment commands for the target application, thereby adjusting the audio signals of each singing group to produce a different auditory experience. It can also generate unique audio works by the conductor, bringing a brand-new choral experience and increasing the fun and participation of the application.

[0298] To better explain the embodiments of this application, the following describes an audio processing method provided by the embodiments of this application in conjunction with a specific implementation scenario. The process of this method can be executed by the interaction between the terminal device and the server, including the following steps, such as... Figure 16 As shown:

[0299] Step 1601: In response to the mode selection operation triggered in the target application, the terminal device sets the singing group mode to the first group mode.

[0300] Step 1602: The terminal device sends the first packet mode it has set to the server.

[0301] Step 1603: In response to the role selection operation triggered by the first participating object in the target application, the terminal device sets the role of the first participating object to the command role.

[0302] Step 1604: The terminal device sends the configured command role to the server.

[0303] Step 1605: The server receives compressed audio data from each of the multiple second participating objects.

[0304] Step 1606: The server decodes each compressed audio data to obtain the corresponding real-time audio data.

[0305] Step 1607: For each second participant, the server extracts sound features based on historical audio data and real-time audio data.

[0306] Step 1608: The server divides multiple second participants into multiple singing object groups according to sound characteristics and the first grouping mode.

[0307] Step 1609: For each singing group, the server performs mixing processing on the real-time audio data of each second participating object in the group to obtain the original mixed audio.

[0308] Step 1610: In response to the first participating object, the terminal device acquires a video segment of the first participating object for an audio adjustment operation triggered for at least one singing object group.

[0309] Step 1611: The terminal device performs gesture recognition and lip-reading recognition on the video clip to obtain audio adjustment instructions for at least one singing group.

[0310] Step 1612: The terminal device sends the obtained audio adjustment command to the server.

[0311] Step 1613: The server adjusts the original mixed audio of the singing object group according to the audio adjustment instructions to obtain the corresponding target mixed audio.

[0312] In step 1614, the server mixes the obtained target mixed audio with the original mixed audio of other singing groups to obtain choral audio.

[0313] Step 1615: The server sends the chorus audio to the terminal device.

[0314] For example, the first interface has three display areas: the left side, the right side, and the middle side. The positional relationship "left hand outside the torso area" is bound to the left side of the first interface, the positional relationship "right hand outside the torso area" is bound to the right side of the first interface, and the positional relationship "left or right hand inside the torso area" is bound to the middle side of the first interface.

[0315] The gesture "slide palm down" corresponds to the audio adjustment command: lower pitch; the gesture "slide palm up" corresponds to the audio adjustment command: raise pitch. The lip-sync gesture "from closed to open" corresponds to the audio adjustment command: increase volume. The lip-sync gesture "from open to closed" corresponds to the audio adjustment command: lower volume.

[0316] During the chorus, see Figure 5 The male voice group 501 is displayed in the left area of ​​the first interface, the female voice group 502 is displayed in the right area of ​​the first interface, and the children's voice group 503 is displayed in the middle area of ​​the first interface 700.

[0317] When it is detected that participant A's left hand is outside the torso area and the left hand gesture is "palm up sliding", participant A's right hand is outside the torso area and the right hand gesture is "palm down sliding", and participant A's mouth shape is "mouth from closed to open", the pitch of male voice group 501 is increased, the pitch of female voice group 502 is decreased, and the volume of male voice group 501, female voice group 502 and children's voice group 503 is increased; then the adjusted audio of male voice group 501, female voice group 502 and children's voice group 503 are mixed to obtain the chorus audio.

[0318] In this embodiment, the first participant takes on a completely new role as a conductor in the choral interaction, allowing the first participant to experience a different interactive effect. Secondly, the body movements of the participants are collected by a local device and converted into preset audio adjustment instructions for the target application, so as to adjust the audio signals of each singing group, thereby producing a different auditory experience. It can also generate unique audio works by the conductor, bringing a brand-new choral experience and increasing the fun and participation of the application.

[0319] Based on the same technical concept, this application provides a schematic diagram of the structure of an audio processing device, such as... Figure 17 As shown, the audio processing device 1700 includes:

[0320] Display module 1701 is used to respond to the first participating object and to display the first interface of the chorus virtual room under the commanding role in the target application of the commanding role. The first interface includes: multiple singing object groups, each singing object group including: at least one second participating object that accesses the chorus virtual room as a singing role.

[0321] The display module 1701 is further configured to respond to the first participating object and, in response to an audio adjustment operation triggered for at least one of the singing object groups, play choral audio in the first interface; wherein, the choral audio is obtained by adjusting the original mixed audio of each of the at least one singing object group separately, and then using at least one target mixed audio obtained to mix with the original mixed audio of each of the other singing object groups.

[0322] Optionally, it also includes a setting module 1702;

[0323] The setting module 1702 is specifically used for:

[0324] In response to the role selection operation triggered by the first participating object in the target application, the role of the first participating object is set to the command role.

[0325] Optionally, the setting module 1702 is further configured to:

[0326] In response to the role-switching operation triggered by the first participant in the first interface, the role of the first participant is switched to the singing role, and the second interface of the chorus virtual room is displayed under the singing role; or,

[0327] In response to the role switching operation triggered by the first participant in the first interface, the role of the first participant is switched to the audience role, and the third interface of the chorus virtual room under the audience role is displayed.

[0328] Optionally, the multiple groups of singing objects are divided according to a preset first grouping pattern;

[0329] The setting module 1702 is also used for:

[0330] In response to a mode selection operation triggered by the first participating object in the target application, the singing group mode is set to the first group mode.

[0331] Optionally, the setting module 1702 is specifically used for:

[0332] For each second participant who accesses the virtual chorus room as a singing role, the following steps are performed: extracting the vocal features of the second participant from its historical singing data; and obtaining the group category corresponding to the second participant in the first grouping mode based on the vocal features of the second participant.

[0333] Based on the grouping categories of the multiple second participants who access the virtual chorus room, the multiple second participants are divided into multiple singing object groups in the first grouping mode.

[0334] Optionally, the setting module 1702 is further configured to:

[0335] In response to the mode switching operation triggered by the first participating object in the target application, the singing group mode is switched to the second group mode, and multiple singing object groups in the second group mode are displayed in the first interface.

[0336] Optionally, it also includes a processing module 1703;

[0337] The processing module 1703 is specifically used for:

[0338] Obtain the body movements of the first participating object;

[0339] Based on the physical movements, at least one audio adjustment instruction is obtained for each of the singing groups;

[0340] According to the audio adjustment instructions corresponding to each singing object group, the original mixed audio corresponding to each singing object group is adjusted to obtain the corresponding target mixed audio;

[0341] The obtained target mixed audio is mixed with the original mixed audio of each of the other singing groups to obtain the chorus audio.

[0342] Optionally, the body movements include: hand gestures;

[0343] The processing module 1703 is specifically used for:

[0344] Collect a video segment of the first participating object, the video segment comprising multiple frame images;

[0345] Identify at least one set of joints for each hand in each frame image;

[0346] Perform the following steps for each hand:

[0347] Based on the obtained set of multiple joint points of a hand, the motion features of the hand are extracted;

[0348] Based on the motion characteristics of the hand, the hand gesture is obtained.

[0349] Optionally, the processing module 1703 is specifically used for:

[0350] For at least one of the frame images, the following steps are performed: based on the set of joints of the hand in the frame image and the torso region identified from the frame image, the positional relationship between the hand and the torso region is obtained;

[0351] Based on at least one of the obtained positional relationships, at least one group of the singing objects controlled by the hand is determined;

[0352] Based on the hand gesture, query the preset gesture mapping data to obtain audio adjustment instructions for at least one of the singing object groups.

[0353] Optionally, the body movements include: lip movements;

[0354] The processing module 1703 is specifically used for:

[0355] Collect a video segment of the first participating object, the video segment comprising multiple frame images;

[0356] The lip region is segmented from each of the frame images;

[0357] Based on the segmented multiple lip regions, lip movement features are extracted;

[0358] Based on the lip movement characteristics, mouth shape actions are obtained.

[0359] Optionally, the processing module 1703 is specifically used for:

[0360] Based on the lip movements, preset lip mapping data is queried to obtain audio adjustment instructions for at least one of the singing object groups.

[0361] In this embodiment, the first participant triggers the chorus participation operation in the target application as a commander, so that the target application displays the first interface of the chorus virtual room under the commander's role. In this way, the first participant can command this new role to participate in the chorus interaction, so that the first participant can experience a different interactive effect.

[0362] Secondly, in response to the first participating object, for the audio adjustment operation triggered by at least one singing object group, the original mixed audio of each of the at least one singing object group is adjusted separately, and the obtained target mixed audio is mixed with the original mixed audio of each of the other singing object groups to obtain the chorus audio. In this way, the first participating object can adjust the mixed audio of each singing object group according to the actual situation, effectively avoiding some audio signals with large volume but low quality from becoming the core sound part of the mixed signal, thereby effectively improving the quality of the chorus audio and thus improving the chorus effect.

[0363] In addition, each participant can adjust the choral audio according to their own preferences, which makes the content of the choral audio richer and more interesting, while increasing the participants' sense of participation in the choral interaction.

[0364] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0365] Based on the same technical concept, embodiments of this application provide a computer device, which can be... Figure 1 The terminal devices and / or servers shown, such as Figure 18 As shown, it includes at least one processor 1801 and a memory 1802 connected to at least one processor. In this embodiment, the specific connection medium between the processor 1801 and the memory 1802 is not limited. Figure 18 Taking the connection between processor 1801 and memory 1802 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0366] In this embodiment of the application, the memory 1802 stores instructions that can be executed by at least one processor 1801. By executing the instructions stored in the memory 1802, at least one processor 1801 can perform the steps of the above-described audio processing method.

[0367] As an example, the processor 1801 responds to the first participating object to display a first interface of the chorus virtual room under the commanding role in the target application to direct the chorus participation operation triggered by the commanding role. The first interface includes: multiple singing object groups, each singing object group including: at least one second participating object that accesses the chorus virtual room as a singing role.

[0368] In response to an audio adjustment operation triggered by a first participating object for at least one of the singing object groups, the processor 1801 plays choral audio in a first interface; wherein, the choral audio is obtained by adjusting the original mixed audio of each of the at least one singing object group separately, and then mixing the obtained at least one target mixed audio with the original mixed audio of each of the other singing object groups; the choral audio is stored in memory 1802.

[0369] The processor 1801 is the control center of the computer device. It can connect to various parts of the computer device through various interfaces and lines. By running or executing instructions stored in the memory 1802 and calling data stored in the memory 1802, it can realize the chorus function under the direction of a conductor. Optionally, the processor 1801 may include one or more processing units. The processor 1801 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1801. In some embodiments, the processor 1801 and the memory 1802 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0370] The processor 1801 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0371] Memory 1802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 1802 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 1802 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer device, but is not limited thereto. In the embodiments of this application, memory 1802 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0372] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the above-described audio processing method.

[0373] Based on the same inventive concept, this application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed by a computer device, cause the computer device to perform the steps of the above-described audio processing method.

[0374] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0375] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer apparatus or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0376] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer device or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0377] These computer program instructions may also be loaded onto a computer device or other programmable data processing equipment to cause a series of operational steps to be performed on the computer device or other programmable equipment to produce a process implemented by the computer device, thereby providing instructions that execute on the computer device or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0378] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0379] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An audio processing method, characterized in that, include: In response to the first participating object, the choral participation operation triggered by the command role in the target application is displayed, and the first interface of the choral virtual room is displayed under the command role. The first interface includes: multiple singing object groups, each singing object group including: at least one second participating object that accesses the choral virtual room as a singing role. In response to the first participating object, for an audio adjustment operation triggered for at least one of the singing object groups, the chorus audio is played in the first interface; wherein, the chorus audio is obtained by adjusting the original mixed audio of each of the at least one singing object group, and then mixing the obtained target mixed audio with the original mixed audio of each of the other singing object groups.

2. The method as described in claim 1, characterized in that, The response to the first participating object, before displaying the chorus participation operation triggered by the commanding role in the target application, and before showing the chorus virtual room under the commanding role's first interface, also includes: In response to the role selection operation triggered by the first participating object in the target application, the role of the first participating object is set to the command role.

3. The method as described in claim 1, characterized in that, The virtual chorus room, following the first interface under the conductor role, also includes: In response to the role-switching operation triggered by the first participant in the first interface, the role of the first participant is switched to the singing role, and the second interface of the chorus virtual room is displayed under the singing role; or, In response to the role switching operation triggered by the first participant in the first interface, the role of the first participant is switched to the audience role, and the third interface of the chorus virtual room under the audience role is displayed.

4. The method as described in claim 1, characterized in that, The multiple singing groups are divided according to a preset first grouping pattern; The response to the first participating object, before displaying the chorus participation operation triggered by the commanding role in the target application, and before showing the chorus virtual room under the commanding role's first interface, also includes: In response to a mode selection operation triggered by the first participating object in the target application, the singing group mode is set to the first group mode.

5. The method as described in claim 4, characterized in that, The multiple singing groups in the first grouping mode are obtained using the following division method: For each second participant who accesses the virtual chorus room as a singing role, the following steps are performed: extracting the vocal features of the second participant from its historical singing data; and obtaining the group category corresponding to the second participant in the first grouping mode based on the vocal features of the second participant. Based on the grouping categories of the multiple second participants who access the virtual chorus room, the multiple second participants are divided into multiple singing object groups in the first grouping mode.

6. The method as described in claim 4, characterized in that, The virtual chorus room, following the first interface under the conductor role, also includes: In response to the mode switching operation triggered by the first participating object in the target application, the singing group mode is switched to the second group mode, and multiple singing object groups in the second group mode are displayed in the first interface.

7. The method as described in any one of claims 1 to 6, characterized in that, The choral audio was obtained using the following method: Obtain the body movements of the first participating object; Based on the physical movements, at least one audio adjustment instruction is obtained for each of the singing groups; According to the audio adjustment instructions corresponding to each singing object group, the original mixed audio corresponding to each singing object group is adjusted to obtain the corresponding target mixed audio; The obtained target mixed audio is mixed with the original mixed audio of each of the other singing groups to obtain the chorus audio.

8. The method as described in claim 7, characterized in that, The body movements include: hand gestures; obtaining the body movements of the first participating object includes: Collect a video segment of the first participating object, the video segment comprising multiple frame images; Identify at least one set of joints for each hand in each frame image; Perform the following steps for each hand: Based on the obtained set of multiple joint points of a hand, the motion features of the hand are extracted; Based on the motion characteristics of the hand, the hand gesture is obtained.

9. The method as described in claim 8, characterized in that, The step of obtaining audio adjustment instructions for at least one of the singing groups based on the body movements includes: For at least one of the frame images, the following steps are performed: based on the set of joints of the hand in the frame image and the torso region identified from the frame image, the positional relationship between the hand and the torso region is obtained; Based on at least one of the obtained positional relationships, at least one group of the singing objects controlled by the hand is determined; Based on the hand gesture, query the preset gesture mapping data to obtain audio adjustment instructions for at least one of the singing object groups.

10. The method as described in claim 7, characterized in that, The body movements include: lip movements; acquiring the body movements of the first participant includes: Collect a video segment of the first participating object, the video segment comprising multiple frame images; The lip region is segmented from each of the frame images; Based on the segmented multiple lip regions, lip movement features are extracted; Based on the lip movement characteristics, mouth shape actions are obtained.

11. The method as described in claim 10, characterized in that, The step of obtaining audio adjustment instructions for at least one of the singing groups based on the body movements includes: Based on the lip movements, preset lip mapping data is queried to obtain audio adjustment instructions for at least one of the singing object groups.

12. An audio processing device, characterized in that, include: The display module is used to respond to the first participating object and to display the first interface of the chorus virtual room under the commanding role in the target application in response to the chorus participation operation triggered by the commanding role. The first interface includes: multiple singing object groups, each singing object group including: at least one second participating object that accesses the chorus virtual room as a singing role. The display module is further configured to respond to the first participating object and, in response to an audio adjustment operation triggered for at least one of the singing object groups, play choral audio in the first interface; wherein, the choral audio is obtained by adjusting the original mixed audio of each of the at least one singing object group separately, and then mixing the obtained target mixed audio with the original mixed audio of each of the other singing object groups.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, It stores a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of any one of claims 1 to 11.

15. A computer program product, characterized in that, The computer program product includes a computer program stored on a computer-readable storage medium, the computer program including program instructions that, when executed by a computer device, cause the computer device to perform the steps of the method according to any one of claims 1-11.

Images