Audio processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an audio processing method and device, equipment and a storage medium, and the method comprises the steps: obtaining background audio data and far-end audio data in an audio and video call, carrying out the voice detection of the far-end audio data, obtaining a voice detection result, determining whether the far-end audio data comprises an effective voice signal or not according to the voice detection result, and obtaining an energy value of the far-end audio data and an energy value of the background audio data, and adjusting the energy of the background audio data and / or the energy of the far-end audio data according to the voice detection result, the energy value of the far-end audio data and the energy value of the background audio data. And enabling the energy value of the adjusted background audio data and the energy value of the adjusted far-end audio data to meet a preset playing condition. Therefore, the volume of the background audio data or the volume of the far-end audio data can be automatically adjusted as required according to whether the far-end audio data comprises the effective voice signal or not, and tedious manual operation is avoided.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to an audio processing method, apparatus, device, and storage medium. Background Technology

[0002] With the development of the internet, audio and video calls, as a convenient and instant communication method, have been widely used in many fields. Audio and video calls can be applied in scenarios such as live streaming, collaborative gaming, voice chat, and online meetings. In practical use, background music (BGM) is often added during audio and video calls. If the volume of the background music is similar to the volume of the remote user, it can lead to difficulty in clearly hearing the remote user's voice.

[0003] The current solution is to provide the ability to adjust the volume of background music and the volume of the remote user's voice separately, allowing the user to manually adjust either the background music volume or the remote user's volume. However, in reality, different background music tracks have different volumes, and when the distance between the remote user and the microphone is not fixed, causing frequent volume changes, the user needs to repeatedly adjust the volume, which is cumbersome and results in a poor user experience. Summary of the Invention

[0004] This application provides an audio processing method, apparatus, device, and storage medium that can automatically adjust the volume of background audio data or remote audio data as needed based on whether the remote audio data includes a valid voice signal, avoiding tedious manual operation.

[0005] In a first aspect, embodiments of this application provide an audio processing method, including:

[0006] Acquire background audio data and remote audio data during audio and video calls;

[0007] Speech detection is performed on the remote audio data to obtain a speech detection result, wherein the speech detection result indicates whether the remote audio data includes a valid speech signal;

[0008] Obtain the energy value of the remote audio data;

[0009] Obtain the energy value of the background audio data;

[0010] Based on the speech detection results, the energy value of the far-end audio data, and the energy value of the background audio data, the energy of the background audio data and / or the energy of the far-end audio data are adjusted so that the energy values ​​of the adjusted background audio data and the far-end audio data meet preset playback conditions.

[0011] Secondly, embodiments of this application provide an audio processing apparatus, including:

[0012] The first acquisition module is used to acquire background audio data and remote audio data in audio and video calls;

[0013] The voice detection module is used to perform voice detection on the remote audio data and obtain a voice detection result, wherein the voice detection result is whether the remote audio data includes a valid voice signal.

[0014] The second acquisition module is used to acquire the energy value of the remote audio data;

[0015] The second acquisition module is further configured to: acquire the energy value of the background audio data;

[0016] The adjustment module is used to adjust the energy of the background audio data and / or the energy of the far-end audio data based on the speech detection result, the energy value of the far-end audio data, and the energy value of the background audio data, so that the energy values ​​of the adjusted background audio data and the far-end audio data meet preset playback conditions.

[0017] Optionally, the voice detection module is used for:

[0018] The remote audio data is parsed to obtain multiple audio data frames, and multiple audio data frames of a preset period are read from the multiple audio data frames;

[0019] For each audio data frame in the multiple audio data frames of the preset period, determine whether the audio data frame contains the sound of the target object;

[0020] The speech detection result is determined based on the ratio of the number of audio data frames containing the sound of the target object within the preset period to the total number of audio data frames within the preset period and a first threshold.

[0021] Optionally, the voice detection module is specifically used for:

[0022] If the ratio of the number of audio data frames containing the sound of the target object within the preset period to the total number of audio data frames within the preset period is greater than or equal to the first threshold, then it is determined that the preset period includes a valid voice signal, and it is determined that the far-end audio data includes a valid voice signal.

[0023] If the ratio of the number of audio data frames containing the sound of the target object within the preset period to the total number of audio data frames within the preset period is less than the first threshold, then it is determined that the preset period does not include valid voice signals.

[0024] Continue reading multiple audio data frames for a preset period from the multiple audio data frames, and detect whether there is a valid voice signal within the preset period, until it is detected that there is no valid voice signal within Q consecutive preset periods, then determine that the far-end audio data does not contain a valid voice signal, where Q is a preset positive integer.

[0025] Optionally, the voice detection module is specifically used for:

[0026] The audio data frame is converted into a frequency domain signal, and the proportion of energy in the frequency domain signal below a preset frequency to the total energy of the audio data frame is calculated.

[0027] If the percentage is greater than a preset threshold, then the audio data frame is determined to contain the sound of the target object.

[0028] Optionally, the second acquisition module is used for:

[0029] The maximum value among the audio values ​​corresponding to the remote audio data is determined as the energy value of the remote audio data, where the audio value corresponding to the remote audio data is a string of K bits, and K is a positive integer; or, the average value of the audio values ​​corresponding to the remote audio data is determined as the energy value of the remote audio data; or, the median value of the audio values ​​corresponding to the remote audio data is determined as the energy value of the remote audio data.

[0030] The maximum value among the audio values ​​corresponding to the background audio data is determined as the energy value of the background audio data, wherein the audio values ​​corresponding to the background audio data are a string of K bits; or, the average value of the audio values ​​corresponding to the background audio data is determined as the energy value of the background audio data; or, the median value of the audio values ​​corresponding to the background audio data is determined as the energy value of the background audio data.

[0031] Optionally, the adjustment module is used for:

[0032] If the speech detection result indicates that the far-end audio data includes a valid speech signal, when the ratio of the energy value of the far-end audio data to the energy value of the background audio data is less than a first preset threshold, the energy of the far-end audio data is increased or the energy of the background audio data is decreased until the ratio of the energy value of the far-end audio data to the energy value of the background audio data after adjustment is greater than the first preset threshold.

[0033] If the speech detection result indicates that the far-end audio data does not include a valid speech signal, when the ratio of the energy value of the background audio data to the energy value of the far-end audio data is less than a second preset threshold, the energy of the far-end audio data is reduced until the ratio of the energy value of the background audio data to the energy value of the adjusted far-end audio data is greater than the second preset threshold.

[0034] Optionally, the adjustment module is used for:

[0035] If the speech detection result indicates that the far-end audio data includes a valid speech signal, when the difference between the energy value of the far-end audio data and the energy value of the background audio data is less than a third preset threshold, the energy of the far-end audio data is increased or the energy of the background audio data is decreased until the difference between the energy value of the adjusted far-end audio data and the energy value of the background audio data is greater than the third preset threshold.

[0036] If the speech detection result indicates that the far-end audio data does not include a valid speech signal, when the difference between the energy value of the background audio data and the energy value of the far-end audio data is less than a fourth preset threshold, the energy of the far-end audio data is reduced until the difference between the energy value of the background audio data and the energy value of the adjusted far-end audio data is greater than the fourth preset threshold.

[0037] Optionally, the first acquisition module is used for:

[0038] The background audio data can be obtained from the server, or from local storage.

[0039] Receive N remote voice streams from the server, where N is a positive integer;

[0040] If N is greater than 1, the N far-end voice streams are mixed to obtain the far-end audio data; if N = 1, the N far-end voice streams are determined as the far-end audio data.

[0041] Optionally, the first acquisition module is specifically used for:

[0042] The audio values ​​corresponding to the N remote voice streams are superimposed to obtain the remote audio data.

[0043] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the method of the first aspect.

[0044] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer program, cause the computer to perform the method as described in the first aspect.

[0045] Fifthly, embodiments of this application provide a computer program product containing instructions that, when executed on a computer, cause the computer to perform the method as described in the first aspect.

[0046] In summary, in this embodiment, by acquiring background audio data and remote audio data from an audio / video call, before mixing the acquired background audio data and remote audio data, the remote audio data is first detected to determine whether it contains a valid voice signal, obtaining a voice detection result. Then, the energy values ​​of the remote audio data and the background audio data are acquired. Based on the voice detection result, the acquired energy values ​​of the remote audio data and the background audio data, the energy of the background audio data and / or the energy of the remote audio data is adjusted so that the adjusted energy values ​​of the background audio data and the remote audio data meet preset playback conditions. Therefore, the energy of the background audio data or the energy of the remote audio data can be automatically adjusted so that the volume of the background audio data or the remote audio data can be automatically adjusted as needed based on whether the remote audio data contains a valid voice signal. This ensures that the background audio data and the remote audio data are played in mixed streams, reducing mutual interference between them, avoiding tedious manual operations, and improving the user experience. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating an audio live-streaming interaction process.

[0048] Figure 2 A schematic diagram illustrating an implementation scenario of an audio processing method provided in this application.

[0049] Figure 3 A flowchart illustrating an audio processing method provided in an embodiment of this application;

[0050] Figure 4 A schematic diagram of the structure of a speech stream provided in an embodiment of this application;

[0051] Figure 5 A schematic diagram of an audio live-streaming process provided for an embodiment of this application;

[0052] Figure 6 A flowchart illustrating an audio processing method provided in an embodiment of this application;

[0053] Figure 7 This is a schematic diagram of the structure of an audio processing device provided in an embodiment of this application;

[0054] Figure 8This is a schematic block diagram of the electronic device 300 provided in the embodiments of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the embodiments of this application.

[0056] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the present application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0057] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. In embodiments of the invention, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0058] The audio processing method provided in this application can be applied to various scenarios such as streaming media, live broadcasting, game collaboration, voice chat, and online conferencing, and can be specifically applied to voice co-op applications in these scenarios.

[0059] It should be noted that the application scenarios described above are for illustrative purposes only and are not intended to limit the scope of this application. In specific implementations, the technical solutions provided in the embodiments of this application can be flexibly applied according to actual needs.

[0060] To facilitate understanding of the embodiments of this application, the relevant concepts involved in the embodiments of this application will first be introduced:

[0061] Voice chat: refers to the act of two or more people communicating in real time on an online platform by turning on their microphones. It is an instant and interactive communication method that can enhance the sense of participation and communication effectiveness among users.

[0062] Audio frame: A fixed-length segment of audio data, typically used in digital audio processing.

[0063] The energy of audio data refers to the volume of the audio signal, which carries energy that can be represented and measured in various ways. In digital audio processing, audio data is typically stored in PCM (Pulse Code Modulation) format, containing a series of samples, each representing the amplitude of the audio signal at a specific moment. The energy distribution of audio data reflects the dynamic range and clarity of the audio signal. Higher energy audio signals generally mean higher clarity and better sound quality.

[0064] Remote audio data refers to the audio data received from the other end of a voice chat session from the server. For users in a voice chat session, such as user A and user B, user B's voice data is user A's remote audio data, and user A's voice data is user B's remote audio data.

[0065] Figure 1 This is a flowchart illustrating an audio live-streaming interaction process, such as... Figure 1 As shown, taking a live chat between two users (User A and User B) as an example, when User A and User B connect, User A is a remote user of User B, and User B is a remote user of User A. User A's terminal device A captures User A's audio and sends it to the server. Terminal device A then downloads User A's audio from the server, allowing User A to hear their own voice. The same process applies to User B.

[0066] User A's terminal device A downloads user B's audio from the server, which is user A's remote audio. Terminal device A downloads background music from the music library. The downloaded background music and remote audio are mixed and sent to terminal device A's player for playback. At this time, user A can hear the background music and the voice of the remote user.

[0067] Similarly, user B's terminal device B downloads user A's audio from the server, which is user B's remote audio. Terminal device B downloads background music from the music library, and the downloaded background music and remote audio are mixed and sent to terminal device B's player for playback. At this time, user B can hear the background music and the voice of the remote user.

[0068] As can be seen from the above process, when background music and remote audio are mixed, if the volume of the background music is greater than the volume of the remote audio, the user cannot hear the voice of the remote user clearly. When the remote audio does not contain human voices but contains ambient sounds, the sound of the background music will also be affected when the background music and remote audio are mixed.

[0069] In related technologies, the user manually adjusts the volume of the background music or the volume of the remote user. This requires the user to repeatedly adjust the volume, which is cumbersome and results in a poor user experience. Moreover, when changes in the volume of the background music or the volume of the remote user cause the remote user's voice to be suppressed by the background music, by the time the user notices and starts manually adjusting the volume, some of the remote user's voice has already been affected, causing the user to miss some of the remote user's voice content.

[0070] To address this issue, this application proposes an audio processing method. During an audio / video call, before mixing the acquired background audio data and remote audio data, the remote audio data is first detected to determine if it contains a valid voice signal. A voice detection result is obtained. Based on this result, the energy value of the acquired remote audio data, and the energy value of the background audio data, the energy of the background audio data and / or the energy of the remote audio data is adjusted so that the adjusted energy values ​​of the background audio data and the remote audio data meet preset playback conditions. This allows for automatic adjustment of the energy of the background audio data or the remote audio data, enabling the volume of either the background audio data or the remote audio data to be automatically adjusted as needed based on whether the remote audio data contains a valid voice signal. This reduces mutual interference between the background audio data and the remote audio data during mixed playback, avoids cumbersome manual operations, and improves the user experience.

[0071] The implementation scenarios of the embodiments of this application are described below.

[0072] For example, Figure 2 This is a schematic diagram illustrating an implementation scenario of an audio processing method provided in an embodiment of this application, such as... Figure 2As shown, the implementation scenario of this application involves server 1, terminal device 2, and terminal device 3. Terminal device 2 and terminal device 3 can communicate with server 1 via a communication network. The communication network can be an intranet, the Internet, Global System for Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, voice communication network, or other wireless or wired networks.

[0073] In some possible implementations, terminal device 2 and terminal device 3 refer to a type of device with rich human-computer interaction methods, internet access capabilities, typically running various operating systems, and possessing strong processing capabilities. Terminal devices can be smartphones, tablets, laptops, desktop computers, or smartwatches, but are not limited to these. Optionally, in this embodiment, terminal device 2 or terminal device 3 may have various applications installed, such as video applications, instant messaging applications, live streaming applications, online conferencing applications, etc.

[0074] In some possible implementations, terminal devices 2 and 3 include, but are not limited to: desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be devices equipped with cameras and display devices, such as smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can be devices equipped with cameras and display devices, such as smartwatches, smart bracelets, and head-mounted devices. Terminal devices are often equipped with display devices, which can also be monitors, displays, touchscreens, etc., and touchscreens can also be touchscreens, touch panels, etc.

[0075] In some embodiments, the aforementioned server 1 may be one or more servers. When there are multiple servers, at least two servers are used to provide different services, and / or at least two servers are used to provide the same service, such as providing the same service in a load-balanced manner. This application embodiment does not limit this. The aforementioned server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server may also become a node in a blockchain.

[0076] It should be noted that the implementation environment of this application embodiment includes, but is not limited to, Figure 2 As shown.

[0077] In one embodiment, terminal device 2 conducts audio and video calls with peer terminal device 3 through a server, such as voice chat or online conferencing. During voice communication, the audio processing method provided in this application embodiment is specifically implemented, for example... Figure 2Terminal device 2 acquires background audio data and remote audio data from an audio / video call. It performs voice detection on the remote audio data to obtain a voice detection result, which indicates whether the remote audio data contains a valid voice signal. It also acquires the energy values ​​of the remote audio data and the background audio data. Based on the voice detection result, the energy values ​​of the remote audio data and the background audio data, it adjusts the energy of the background audio data and / or the energy of the remote audio data to ensure that the adjusted energy values ​​meet preset playback conditions. Specifically, the background audio data and remote audio data can be acquired from a server, or the remote audio data can be acquired from the server, while the background audio data can be acquired from the internal storage of terminal device 2. Optionally, in one embodiment, the executing entity for the audio processing method provided in this application embodiment can also be a server. Specifically, the server acquires background audio data and remote audio data, performs speech detection on the remote audio data to obtain a speech detection result, which indicates whether the remote audio data includes a valid speech signal. The server then acquires the energy values ​​of the remote audio data and the background audio data. Based on the speech detection result, the energy values ​​of the remote audio data and the background audio data, the server adjusts the energy of either the background audio data or the remote audio data so that the adjusted energy values ​​meet preset playback conditions. The adjusted background audio data and remote audio data are then sent to terminal device 2, where terminal device 2 mixes the adjusted background audio data and remote audio data and plays it through a player. In this case, the server can acquire the background audio data from terminal device 2 or from its storage, and the remote audio data can be acquired from the server's storage.

[0078] It is understood that in the specific implementation of this application, data related to user information (such as remote audio data) is involved. When the method of this application embodiment is applied to a specific product or technology, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0079] The technical solutions of the embodiments of this application will be described in detail below through some examples. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0080] Figure 3 This is a flowchart illustrating an audio processing method provided in an embodiment of this application. The execution entity in this embodiment can be a terminal device or a server. This embodiment uses a terminal device as an example for explanation. Figure 3 As shown, the method may include:

[0081] S101. Obtain background audio data and remote audio data from the audio / video call.

[0082] Specifically, when a terminal device conducts an audio or video call (such as a voice chat), it first acquires background audio data and remote audio data. During the actual voice chat, the user adds some background music (i.e., background audio data). The background audio data can be stored in the terminal device's local storage or in a server. The remote audio data is the audio data received from the server from the other end of the voice chat. For users engaging in voice chat, such as two users A and B, user B's voice data is user A's remote audio data, and user A's voice data is user B's remote audio data.

[0083] Optionally, in one implementable manner, acquiring background audio data and remote audio data in S101 can specifically be:

[0084] S1011. Obtain background audio data from the server, or obtain background audio data from local storage.

[0085] S1012. Receive N remote voice streams from the server, where N is a positive integer.

[0086] S1013. If N is greater than 1, mix the N far-end speech streams to obtain far-end audio data. If N = 1, determine the N far-end speech streams as far-end audio data.

[0087] Specifically, in multi-user voice communication scenarios, such as online meetings where there are multiple remote users, it is necessary to first mix the voice streams (audio data) of the multiple remote users into a single voice stream, which is then used to acquire the remote audio data. For single-user voice communication, where there is only one remote user, no mixing is needed; the single remote voice stream is used as the acquired remote audio data.

[0088] Optionally, in one embodiment, the N far-end voice streams are mixed, specifically by superimposing the audio values ​​corresponding to the N far-end voice streams to obtain far-end audio data. Figure 4 This application provides a schematic diagram of the structure of a speech stream, as shown in the embodiment of the present application. Figure 4 As shown, the speech stream consists of a series of 16-bit values, typically more than 3,000 or 4,000 16-bit values, with values ​​ranging from 5,000 to 65,536. Mixing N remote speech streams can be achieved by superimposing the corresponding audio values ​​of the N remote speech streams to obtain the remote audio data. For example, if the speech stream consists of 3,840 16-bit values, superimposing the corresponding 3,840 16-bit values ​​of the N remote speech streams will result in the superimposed speech stream, which is the remote audio data.

[0089] S102. Perform speech detection on the remote audio data to obtain the speech detection result, which indicates whether the remote audio data includes a valid speech signal.

[0090] Specifically, speech detection checks whether the remote audio data includes a valid speech signal, such as human voice information. The following describes a speech detection method in detail.

[0091] Optionally, as an implementable method, S102 performs speech detection on the far-end audio data to obtain the speech detection result, which can be specifically as follows:

[0092] S1021. Parse the remote audio data to obtain multiple audio data frames, and read multiple audio data frames of a preset period from the multiple audio data frames.

[0093] Specifically, audio data frames are generally 20ms or 10ms data frames. Remote audio data may include one audio data frame or multiple audio data frames. If it includes one audio data frame, multiple audio data frames of a preset period are parsed and read in sequence.

[0094] S1022. For each audio data frame in a plurality of audio data frames in a preset period, determine whether the audio data frame contains the sound of the target object.

[0095] Specifically, the target object can be any user, and the goal is to determine whether an audio data frame contains a human voice. For each audio data frame within a preset period, it is determined whether each audio data frame contains the voice of the target object; specifically, it is determined whether that audio data frame contains a human voice, any human voice. For example, audio data frames are typically 20ms or 10ms. Taking a 20ms audio data frame as an example, with a preset period of 200ms, the total number of audio data frames within the preset period is 10. For each of these 10 audio data frames, it is determined whether each audio data frame contains the voice of the target object. Similarly, taking a 100ms audio data frame as an example, with a preset period of 200ms, the total number of audio data frames within the preset period is 20. For each of these 20 audio data frames, it is determined whether each audio data frame contains the voice of the target object.

[0096] Optionally, as an implementable method, in S1022, for each audio data frame in a plurality of audio data frames within a preset period, it is determined whether the audio data frame contains the sound of the target object. Specifically, this can be done as follows:

[0097] S1. Convert the audio data frame into a frequency domain signal and calculate the proportion of energy in the frequency domain signal below the preset frequency to the total energy of the audio data frame.

[0098] S2. If the proportion is greater than the preset threshold, then the audio data frame is determined to contain the sound of the target object.

[0099] Specifically, for each audio data frame, the audio data frame is first converted into a frequency domain signal. The proportion of energy in the frequency domain signal below a preset frequency (e.g., 2kHz) to the total energy of the audio data frame is calculated. If this proportion is greater than a preset threshold, it is determined that the audio data frame contains the sound of the target object. Here, the target object can be any user, and determining that the audio data frame contains the sound of the target object means determining that the audio data frame contains a human voice.

[0100] S1023. Determine the speech detection result based on the ratio of the number of audio data frames containing the sound of the target object within a preset period to the total number of audio data frames within the preset period and a first threshold.

[0101] For example, if the total number of audio data frames in a preset period is 10, and the number of audio data frames containing the sound of the target object in a preset period is 5, then the ratio of the number of audio data frames containing the sound of the target object in a preset period to the total number of audio data frames in a preset period is 1 / 2.

[0102] Optionally, in one implementable manner, S1023 may specifically be:

[0103] S11. If the ratio of the number of audio data frames containing the sound of the target object within a preset period to the total number of audio data frames within the preset period is greater than or equal to a first threshold, then it is determined that the preset period includes a valid voice signal, and it is determined that the remote audio data includes a valid voice signal.

[0104] S12. If the ratio of the number of audio data frames containing the sound of the target object within a preset period to the total number of audio data frames within the preset period is less than a first threshold, then it is determined that the preset period does not include valid voice signals.

[0105] S13. Continue to read multiple audio data frames of a preset period from multiple audio data frames, and detect whether there is a valid voice signal within the preset period, until it is detected that there is no valid voice signal within Q consecutive preset periods, then determine that the far-end audio data does not contain a valid voice signal, where Q is a preset positive integer.

[0106] Specifically, for the first preset period, if the first preset period includes a valid voice signal, then it is determined that the remote audio data includes a valid voice signal; if the first preset period does not include a valid voice signal, then the audio data frame of the second preset period is detected. If the second preset period includes a valid voice signal, then it is determined that the remote audio data includes a valid voice signal; if the second preset period does not include a valid voice signal, then the audio data frame of the third preset period is detected, until it is detected that there is no valid voice signal in Q consecutive preset periods, then it is determined that the remote audio data does not include a valid voice signal, for example, Q is 3 or 5, etc.

[0107] S103, Obtain the energy value of the remote audio data.

[0108] S104. Obtain the energy value of the background audio data.

[0109] Specifically, the energy of audio data refers to the volume corresponding to the played audio, which is the energy carried in the audio signal. This energy can be represented and measured in different ways. The energy distribution of audio data can reflect the dynamic range and clarity of the audio signal. High-energy audio signals usually mean higher clarity and better sound quality. Correspondingly, the energy value of audio data can correspond to the volume level of the played audio. The following is an implementation method for obtaining the energy values ​​of remote audio data and background audio data.

[0110] Optionally, in one implementable manner, S103 may specifically be:

[0111] S1031. The maximum value among the audio values ​​corresponding to the remote audio data is determined as the energy value of the remote audio data.

[0112] Optionally, in one implementable manner, S104 can specifically be:

[0113] S1041. The maximum value among the audio values ​​corresponding to the background audio data is determined as the energy value of the background audio data. The audio values ​​corresponding to the far-end audio data and the audio values ​​corresponding to the background audio data are both K-bit values, where K is a positive integer.

[0114] The energy value of the far-end audio data is used to assess whether the far-end audio data can still be clearly heard after mixing. Here, K is, for example, 16. Figure 4As shown, the speech stream (i.e., audio data) consists of a series of 16-bit values, typically around 3,000 or 4,000 16-bit values, with values ​​ranging from 5,000 to 65,536. When obtaining the energy value of the far-end audio data, the maximum value among the corresponding audio values ​​can be determined as the energy value. For example, if the far-end audio data consists of 3,840 16-bit values, the maximum value among these 3,840 16-bit values ​​can be determined as the energy value. Alternatively, the average value of the corresponding audio values ​​can be determined as the energy value, or the median value can be determined, etc. This embodiment does not impose any limitations on these methods.

[0115] When obtaining the energy value of background audio data, the maximum value among the corresponding audio values ​​can be determined as the energy value of the background audio data. Alternatively, the average value of the corresponding audio values ​​can be determined as the energy value of the background audio data, or the median value of the corresponding audio values ​​can be determined as the energy value of the background audio data, etc. This embodiment does not limit this.

[0116] S105. Based on the speech detection results, the energy value of the far-end audio data, and the energy value of the background audio data, adjust the energy of the background audio data and / or the energy of the far-end audio data so that the adjusted energy values ​​of the background audio data and the far-end audio data meet the preset playback conditions.

[0117] Specifically, after obtaining the speech detection result, the energy value of the far-end audio data, and the energy value of the background audio data, the energy of the background audio data and / or the energy of the far-end audio data can be adjusted. Specifically, at least one of the energy values ​​of the background audio data and the far-end audio data can be adjusted to ensure that the adjusted energy values ​​of the background audio data and the far-end audio data meet preset playback conditions. For example, when the speech detection result indicates that the far-end audio data includes a valid speech signal, the far-end audio data can meet preset playback clarity requirements; when the speech detection result indicates that the far-end audio data does not include a valid speech signal, the background audio data can meet preset playback clarity requirements.

[0118] Optionally, as an implementable approach, in S105, based on the speech detection results, the energy value of the far-end audio data, and the energy value of the background audio data, the energy of the background audio data is adjusted so that the adjusted energy values ​​of the background audio data and the far-end audio data meet preset playback conditions. Specifically, this can be achieved by:

[0119] S1051. If the speech detection result is that the remote audio data includes a valid speech signal, when the ratio of the energy value of the remote audio data to the energy value of the background audio data is less than a first preset threshold, increase the energy of the remote audio data or decrease the energy of the background audio data until the ratio of the energy value of the remote audio data to the energy value of the background audio data after adjustment is greater than the first preset threshold.

[0120] Specifically, when the voice detection result indicates that the remote audio data includes valid voice signals, the remote audio data needs to be protected. If the ratio of the energy value of the remote audio data to the energy value of the background audio data is greater than a first preset threshold, the remote audio data is determined to be unaffected and requires no adjustment. If the ratio is less than the first preset threshold, the remote audio data is determined to be affected by the background audio data, requiring adjustment. Specifically, the energy of the remote audio data can be increased or the energy of the background audio data can be decreased until the adjusted ratio is greater than the first preset threshold. Thus, during voice interaction, when the remote audio data includes human voice data, the system automatically increases the energy of the remote audio data or decreases the energy of the background audio data until the adjusted ratio is greater than the first preset threshold, automatically protecting the remote audio data from the influence of background audio data. This allows the user to clearly hear the remote audio data without manual intervention.

[0121] S1052. If the speech detection result is that the far-end audio data does not include a valid speech signal, when the ratio of the energy value of the background audio data to the energy value of the far-end audio data is less than the second preset threshold, the energy of the far-end audio data is reduced until the ratio of the energy value of the background audio data to the energy value of the adjusted far-end audio data is greater than the second preset threshold.

[0122] Specifically, when the voice detection result indicates that the far-end audio data does not contain valid voice signals, meaning the far-end voice stream does not include human voice, it is necessary to protect the background audio data. If the ratio of the energy value of the background audio data to the energy value of the far-end audio data is greater than a second preset threshold, the background audio data is considered unaffected and requires no adjustment. If the ratio is less than the second preset threshold, adjustment is necessary. Specifically, the energy of the far-end audio data can be reduced until the ratio of the energy value of the background audio data to the adjusted energy value of the far-end audio data exceeds the second preset threshold. This allows for automatic reduction of the far-end audio data's energy during voice interaction when there is no human voice data, protecting the background audio data and ensuring its audio quality.

[0123] Alternatively, as another implementable approach, S105 can specifically be:

[0124] S1051' If the voice detection result is that the far-end audio data includes a valid voice signal, when the difference between the energy value of the far-end audio data and the energy value of the background audio data is less than the third preset threshold, increase the energy of the far-end audio data or decrease the energy of the background audio data until the difference between the energy value of the far-end audio data and the energy value of the background audio data after adjustment is greater than the third preset threshold.

[0125] S1052' If the speech detection result is that the far-end audio data does not include a valid speech signal, when the difference between the energy value of the background audio data and the energy value of the far-end audio data is less than the fourth preset threshold, the energy of the far-end audio data is reduced until the difference between the energy value of the background audio data and the energy value of the adjusted far-end audio data is greater than the fourth preset threshold.

[0126] The audio processing method provided in this embodiment acquires background audio data and remote audio data from an audio / video call. Before mixing the acquired background audio data and remote audio data, it first detects whether the remote audio data contains a valid voice signal, obtaining a voice detection result. Then, it acquires the energy values ​​of the remote audio data and the background audio data. Based on the voice detection result, the acquired energy values ​​of the remote audio data, and the background audio data, it adjusts the energy of the background audio data and / or the energy of the remote audio data so that the adjusted energy values ​​of the background audio data and the remote audio data meet preset playback conditions. Therefore, the energy of the background audio data or the energy of the remote audio data can be automatically adjusted to ensure that the volume of the background audio data or the remote audio data can be automatically adjusted as needed based on whether the remote audio data contains a valid voice signal. This reduces mutual interference between the background audio data and the remote audio data during mixed playback, avoids cumbersome manual operation, and improves the user experience.

[0127] The following is combined Figure 5 and Figure 6 The audio processing process will be explained in detail using a specific embodiment.

[0128] Figure 5 This application provides a schematic diagram of an audio live-streaming process, as illustrated in the embodiments of this application. Figure 5 As shown, compared to Figure 1 The structure shown in this application embodiment includes an audio protection module, which is used to perform... Figure 3 Or as follows Figure 6The audio processing method shown, through the newly added audio protection module, eliminates the need for user operation to adjust the volume. It can automatically adjust the volume of the remote voice or the background music when the user is on a voice call and background music is playing. This ensures that the user's reception of the remote user's voice is not affected by the background music, allowing the user to clearly hear the remote user's voice data. At the same time, it can also protect the background music when there is no remote voice, ensuring the audio quality of the background music.

[0129] Specifically, such as Figure 5 As shown, for terminal device A, the terminal device A used by user A downloads user B's audio from the server, which is user A's remote audio. Terminal device A downloads background music from the music library, which can be the music library stored on the terminal device. After obtaining the background music and remote audio, terminal device A inputs them to the audio protection module for audio processing. The processed background music and remote audio are mixed and sent to the player of terminal device A for playback. At this time, user A can hear the background music and the voice of the remote user.

[0130] For terminal device B, the terminal device B used by user B downloads user A's audio from the server, which is user B's remote audio. Terminal device B downloads background music from the music library, which can be the music library stored on the terminal device. After obtaining the background music and remote audio, terminal device B inputs them to the audio protection module for audio processing. The processed background music and remote audio are mixed and sent to the player of terminal device B for playback. At this time, user B can hear the background music and the voice of the remote user.

[0131] Figure 6 This is a flowchart illustrating an audio processing method provided in an embodiment of this application. The execution entity in this embodiment can be a terminal device or a server. This embodiment uses a terminal device as an example for explanation. Figure 6 As shown, the method may include:

[0132] S201. Obtain background audio data and remote audio data from the audio / video call.

[0133] Specifically, when a terminal device engages in voice communication, it first acquires background audio data and remote audio data. During actual voice communication, the user adds some background music (i.e., background audio data). The background audio data can be stored in the terminal device's local storage or on a server. The remote audio data is the audio data received from the other end of the voice communication from the server. For users engaging in voice communication, such as two users A and B, user B's voice data is user A's remote audio data, and user A's voice data is user B's remote audio data.

[0134] S202. Perform speech detection on the remote audio data to obtain the speech detection result, which indicates whether the remote audio data includes a valid speech signal.

[0135] Specifically, speech detection checks whether the remote audio data includes a valid speech signal, such as human voice information. The following describes a speech detection method in detail.

[0136] Optionally, as an implementable method, S202 performs speech detection on the remote audio data to obtain the speech detection results, which can specifically be:

[0137] S2021. Parse the remote audio data to obtain multiple audio data frames, and read multiple audio data frames of a preset period from the multiple audio data frames.

[0138] Specifically, audio data frames are generally 20ms or 10ms data frames. Remote audio data may include one audio data frame or multiple audio data frames. If it includes one audio data frame, multiple audio data frames of a preset period are parsed and read in sequence.

[0139] S2022. For each audio data frame in a preset period of multiple audio data frames, determine whether the audio data frame contains the sound of the target object.

[0140] Specifically, the target object can be the user, determining whether the audio data frame contains a human voice.

[0141] Optionally, as an implementable approach, in S2022, for each audio data frame in a plurality of audio data frames within a preset period, it is determined whether the audio data frame contains the sound of the target object. Specifically, this can be done as follows:

[0142] S1. Convert the audio data frame into a frequency domain signal and calculate the proportion of energy in the frequency domain signal below the preset frequency to the total energy of the audio data frame.

[0143] S2. If the proportion is greater than the preset threshold, then the audio data frame is determined to contain the sound of the target object.

[0144] Specifically, for each audio data frame, the audio data frame is first converted into a frequency domain signal, and the proportion of the energy of the part of the frequency domain signal below a preset frequency (e.g., 2kHz) to the total energy of the audio data frame is calculated. If the proportion is greater than a preset threshold, it is determined that the audio data frame contains the sound of the target object.

[0145] S2023. Determine the speech detection result based on the ratio of the number of audio data frames containing the sound of the target object within a preset period to the total number of audio data frames within the preset period and a first threshold.

[0146] For example, if the total number of audio data frames in a preset period is 10, and the number of audio data frames containing the sound of the target object in a preset period is 5, then the ratio of the number of audio data frames containing the sound of the target object in a preset period to the total number of audio data frames in a preset period is 1 / 2.

[0147] Optionally, in one implementable manner, S2023 may specifically be:

[0148] S21. If the ratio of the number of audio data frames containing the sound of the target object within a preset period to the total number of audio data frames within the preset period is greater than or equal to a first threshold, then it is determined that the preset period includes a valid voice signal, and it is determined that the remote audio data includes a valid voice signal.

[0149] S22. If the ratio of the number of audio data frames containing the sound of the target object within a preset period to the total number of audio data frames within the preset period is less than a first threshold, then it is determined that the preset period does not include valid speech signals.

[0150] S23. Continue to read multiple audio data frames of a preset period from multiple audio data frames, and detect whether there is a valid voice signal within the preset period, until it is detected that there is no valid voice signal within Q consecutive preset periods, then determine that the far-end audio data does not contain a valid voice signal, where Q is a preset positive integer.

[0151] Specifically, for the first preset period, if the first preset period includes a valid voice signal, then it is determined that the remote audio data includes a valid voice signal; if the first preset period does not include a valid voice signal, then the audio data frame of the second preset period is detected. If the second preset period includes a valid voice signal, then it is determined that the remote audio data includes a valid voice signal; if the second preset period does not include a valid voice signal, then the audio data frame of the third preset period is detected, until it is detected that there is no valid voice signal in Q consecutive preset periods, then it is determined that the remote audio data does not include a valid voice signal, for example, Q is 3 or 5, etc.

[0152] S203. Obtain the energy value of the remote audio data and the energy value of the background audio data.

[0153] The following is an implementation method for obtaining the energy values ​​of remote audio data and background audio data.

[0154] Optionally, in one implementable manner, S203 may specifically be:

[0155] S2031. The maximum value among the audio values ​​corresponding to the remote audio data is determined as the energy value of the remote audio data.

[0156] S2032. The maximum value among the audio values ​​corresponding to the background audio data is determined as the energy value of the background audio data, wherein the audio value corresponding to the far-end audio data and the audio value corresponding to the background audio data are both K-bit values, where K is a positive integer.

[0157] S204. If the speech detection result is that the far-end audio data includes a valid speech signal, when the ratio of the energy value of the far-end audio data to the energy value of the background audio data is less than a first preset threshold, increase the energy of the far-end audio data or decrease the energy of the background audio data until the ratio of the energy value of the far-end audio data to the energy value of the background audio data after adjustment is greater than the first preset threshold.

[0158] Specifically, when the voice detection result indicates that the remote audio data includes valid voice signals, the remote audio data needs to be protected. If the ratio of the energy value of the remote audio data to the energy value of the background audio data is greater than a first preset threshold, the remote audio data is determined to be unaffected and requires no adjustment. If the ratio is less than the first preset threshold, the remote audio data is determined to be affected by the background audio data, requiring adjustment. Specifically, the energy of the remote audio data can be increased or the energy of the background audio data can be decreased until the adjusted ratio is greater than the first preset threshold. Thus, during voice interaction, when the remote audio data includes human voice data, the system automatically increases the energy of the remote audio data or decreases the energy of the background audio data until the adjusted ratio is greater than the first preset threshold, automatically protecting the remote audio data from the influence of background audio data. This allows the user to clearly hear the remote audio data without manual intervention.

[0159] S205. If the speech detection result is that the far-end audio data does not include a valid speech signal, when the ratio of the energy value of the background audio data to the energy value of the far-end audio data is less than the second preset threshold, the energy of the far-end audio data is reduced until the ratio of the energy value of the background audio data to the energy value of the adjusted far-end audio data is greater than the second preset threshold.

[0160] Specifically, when the voice detection result indicates that the far-end audio data does not contain valid voice signals, meaning the far-end voice stream does not include human voice, it is necessary to protect the background audio data. If the ratio of the energy value of the background audio data to the energy value of the far-end audio data is greater than a second preset threshold, the background audio data is considered unaffected and requires no adjustment. If the ratio is less than the second preset threshold, adjustment is necessary. Specifically, the energy of the far-end audio data can be reduced until the ratio of the energy value of the background audio data to the adjusted energy value of the far-end audio data exceeds the second preset threshold. This allows for automatic reduction of the far-end audio data's energy during voice interaction when there is no human voice data, protecting the background audio data and ensuring its audio quality.

[0161] Alternatively, as another implementable approach, S204 can specifically be:

[0162] S204' If the speech detection result is that the far-end audio data includes a valid speech signal, when the difference between the energy value of the far-end audio data and the energy value of the background audio data is less than the third preset threshold, increase the energy of the far-end audio data or decrease the energy of the background audio data until the difference between the energy value of the far-end audio data and the energy value of the background audio data after adjustment is greater than the third preset threshold.

[0163] Alternatively, as another implementable approach, S205 can specifically be:

[0164] S205' If the voice detection result is that the far-end audio data does not include a valid voice signal, when the difference between the energy value of the background audio data and the energy value of the far-end audio data is less than the fourth preset threshold, the energy of the far-end audio data is reduced until the difference between the energy value of the background audio data and the energy value of the adjusted far-end audio data is greater than the fourth preset threshold.

[0165] The audio processing method provided in this embodiment acquires background audio data and remote audio data from an audio / video call. Before mixing the acquired background audio data and remote audio data, it first detects whether the remote audio data contains a valid voice signal, obtaining a voice detection result. Then, it acquires the energy values ​​of the remote audio data and the background audio data. Based on the voice detection result and the acquired energy values ​​of the remote audio data and the background audio data, it adjusts the energy of the background audio data and / or the energy of the remote audio data so that the adjusted energy values ​​of the background audio data and the remote audio data meet preset playback conditions. Therefore, during an audio / video call, when the remote audio data includes human voice data, it automatically increases the energy of the remote audio data or decreases the energy of the background audio data until the ratio of the adjusted energy value of the remote audio data to the energy value of the background audio data is greater than a first preset threshold, automatically protecting the remote audio data from the influence of the background audio data. This allows the user to clearly hear the remote audio data without manual operation. When the remote audio data does not contain human voice data, it automatically reduces the energy of the remote audio data to protect the background audio data and ensure its audio quality.

[0166] Figure 7 This is a schematic diagram of the structure of an audio processing device provided in an embodiment of this application, as shown below. Figure 7 As shown, the device may include: a first acquisition module 11, a voice detection module 12, a second acquisition module 13, and an adjustment module 14.

[0167] The first acquisition module 11 is used to acquire background audio data and remote audio data in audio and video calls.

[0168] The voice detection module 12 is used to perform voice detection on the remote audio data and obtain the voice detection result, which indicates whether the remote audio data includes a valid voice signal.

[0169] The second acquisition module 13 is used to acquire the energy value of the remote audio data.

[0170] The second acquisition module 13 is also used to acquire the energy value of the background audio data.

[0171] The adjustment module 14 is used to adjust the energy of the background audio data and / or the energy of the far-end audio data according to the speech detection results, the energy value of the far-end audio data, and the energy value of the background audio data, so that the energy value of the adjusted background audio data and the energy value of the far-end audio data meet the preset playback conditions.

[0172] Optionally, the voice detection module 12 is used for:

[0173] Parse the remote audio data to obtain multiple audio data frames, and read multiple audio data frames of a preset period from the multiple audio data frames;

[0174] For each audio data frame in a set period of multiple audio data frames, determine whether the audio data frame contains the sound of the target object;

[0175] The speech detection result is determined based on the ratio of the number of audio data frames containing the sound of the target object within a preset period to the total number of audio data frames within the preset period and a first threshold.

[0176] Optionally, the voice detection module 12 is specifically used for:

[0177] If the ratio of the number of audio data frames containing the sound of the target object within a preset period to the total number of audio data frames within the preset period is greater than or equal to the first threshold, then it is determined that the preset period includes a valid voice signal, and it is determined that the far-end audio data includes a valid voice signal.

[0178] If the ratio of the number of audio data frames containing the sound of the target object within a preset period to the total number of audio data frames within the preset period is less than a first threshold, then it is determined that no valid voice signal is included within the preset period.

[0179] Continue reading multiple audio data frames for a preset period from multiple audio data frames, and detect whether there is a valid voice signal within the preset period, until it is detected that there is no valid voice signal within Q consecutive preset periods, then determine that the far-end audio data does not contain a valid voice signal, where Q is a preset positive integer.

[0180] Optionally, the voice detection module 12 is specifically used for:

[0181] Convert the audio data frame into a frequency domain signal and calculate the proportion of energy in the frequency domain signal below a preset frequency to the total energy of the audio data frame.

[0182] If the proportion is greater than the preset threshold, then the audio data frame is determined to contain the sound of the target object.

[0183] Optionally, the second acquisition module is used for:

[0184] The maximum value among the audio values ​​corresponding to the remote audio data is determined as the energy value of the remote audio data, where the audio value corresponding to the remote audio data is a string of K bits, and K is a positive integer; or, the average value of the audio values ​​corresponding to the remote audio data is determined as the energy value of the remote audio data; or, the median value of the audio values ​​corresponding to the remote audio data is determined as the energy value of the remote audio data.

[0185] The maximum value among the audio values ​​corresponding to the background audio data is determined as the energy value of the background audio data, where the audio values ​​corresponding to the background audio data are a string of K bits; or, the average value of the audio values ​​corresponding to the background audio data is determined as the energy value of the background audio data; or, the median value of the audio values ​​corresponding to the background audio data is determined as the energy value of the background audio data.

[0186] Optionally, adjustment module 14 is used for:

[0187] If the speech detection result indicates that the far-end audio data includes a valid speech signal, and the ratio of the energy value of the far-end audio data to the energy value of the background audio data is less than a first preset threshold, the energy of the far-end audio data is increased or the energy of the background audio data is decreased until the ratio of the energy value of the far-end audio data to the energy value of the background audio data after adjustment is greater than the first preset threshold.

[0188] If the speech detection result indicates that the far-end audio data does not include a valid speech signal, and the ratio of the energy value of the background audio data to the energy value of the far-end audio data is less than the second preset threshold, the energy of the far-end audio data is reduced until the ratio of the energy value of the background audio data to the energy value of the adjusted far-end audio data is greater than the second preset threshold.

[0189] Optionally, adjustment module 14 is used for:

[0190] If the speech detection result indicates that the far-end audio data includes a valid speech signal, and the difference between the energy value of the far-end audio data and the energy value of the background audio data is less than the third preset threshold, increase the energy of the far-end audio data or decrease the energy of the background audio data until the difference between the energy value of the far-end audio data and the energy value of the background audio data after adjustment is greater than the third preset threshold.

[0191] If the speech detection result indicates that the far-end audio data does not include a valid speech signal, and the difference between the energy value of the background audio data and the energy value of the far-end audio data is less than the fourth preset threshold, the energy of the far-end audio data is reduced until the difference between the energy value of the background audio data and the energy value of the adjusted far-end audio data is greater than the fourth preset threshold.

[0192] Optionally, the first acquisition module 11 is used for:

[0193] Retrieve background audio data from the server, or retrieve background audio data from local storage;

[0194] Receive N remote voice streams from the server, where N is a positive integer;

[0195] If N is greater than 1, mix the N far-end speech streams to obtain far-end audio data; if N = 1, determine the N far-end speech streams as far-end audio data.

[0196] Optionally, the first acquisition module 11 is specifically used for:

[0197] The audio values ​​corresponding to the N remote voice streams are superimposed to obtain the remote audio data.

[0198] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 7 The audio processing apparatus shown can execute the method embodiments corresponding to the terminal device or server, and the foregoing and other operations and / or functions of each module in the apparatus are respectively for implementing the method embodiments corresponding to the terminal device or server, which will not be described in detail here for the sake of brevity.

[0199] The audio processing apparatus and audio processing device of the present application embodiments have been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in the present application embodiments can be completed by the integrated logic circuits in the processor hardware and / or by software instructions. The steps of the method disclosed in the present application embodiments can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0200] Figure 8 This is a schematic block diagram of the electronic device 300 provided in the embodiments of this application.

[0201] like Figure 8 As shown, the electronic device 300 may include:

[0202] The system includes a memory 310 and a processor 320. The memory 310 stores computer programs and transfers the program code to the processor 320. In other words, the processor 320 can retrieve and run the computer program from the memory 310 to implement the methods described in the embodiments of this application.

[0203] For example, the processor 320 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0204] In some embodiments of this application, the processor 320 may include, but is not limited to:

[0205] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0206] In some embodiments of this application, the memory 310 includes, but is not limited to:

[0207] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0208] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 310 and executed by the processor 320 to complete the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0209] like Figure 8 As shown, the electronic device may also include:

[0210] Transceiver 330, which can be connected to processor 320 or memory 310.

[0211] The processor 320 can control the transceiver 330 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 330 may include a transmitter and a receiver. The transceiver 330 may further include antennas, and the number of antennas may be one or more.

[0212] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0213] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, this application also provides a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0214] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0215] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0216] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.

[0217] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0218] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be included within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. An audio processing method, characterized in that, include: Acquire background audio data and remote audio data during audio and video calls; Speech detection is performed on the remote audio data to obtain a speech detection result, wherein the speech detection result indicates whether the remote audio data includes a valid speech signal; Obtain the energy value of the remote audio data; Obtain the energy value of the background audio data; Based on the speech detection results, the energy value of the far-end audio data, and the energy value of the background audio data, the energy of the background audio data and / or the energy of the far-end audio data are adjusted so that the energy values ​​of the adjusted background audio data and the far-end audio data meet preset playback conditions.

2. The method according to claim 1, characterized in that, The step of performing speech detection on the remote audio data to obtain speech detection results includes: The remote audio data is parsed to obtain multiple audio data frames, and multiple audio data frames of a preset period are read from the multiple audio data frames; For each audio data frame in the multiple audio data frames of the preset period, determine whether the audio data frame contains the sound of the target object; The speech detection result is determined based on the ratio of the number of audio data frames containing the sound of the target object within the preset period to the total number of audio data frames within the preset period and a first threshold.

3. The method according to claim 2, characterized in that, The step of determining the speech detection result based on the ratio of the number of audio data frames containing the sound of the target object within the preset period to the total number of audio data frames within the preset period and a first threshold includes: If the ratio of the number of audio data frames containing the sound of the target object within the preset period to the total number of audio data frames within the preset period is greater than or equal to the first threshold, then it is determined that the preset period includes a valid voice signal, and it is determined that the far-end audio data includes a valid voice signal. If the ratio of the number of audio data frames containing the sound of the target object within the preset period to the total number of audio data frames within the preset period is less than the first threshold, then it is determined that the preset period does not include valid voice signals. Continue reading multiple audio data frames for a preset period from the multiple audio data frames, and detect whether there is a valid voice signal within the preset period, until it is detected that there is no valid voice signal within Q consecutive preset periods, then determine that the far-end audio data does not contain a valid voice signal, where Q is a preset positive integer.

4. The method according to claim 2, characterized in that, Determining whether each audio data frame in a plurality of audio data frames within the preset period contains the sound of the target object includes: The audio data frame is converted into a frequency domain signal, and the proportion of energy in the frequency domain signal below a preset frequency to the total energy of the audio data frame is calculated. If the percentage is greater than a preset threshold, then the audio data frame is determined to contain the sound of the target object.

5. The method according to claim 1, characterized in that, The process of acquiring the energy value of the remote audio data includes: The maximum value among the audio values ​​corresponding to the remote audio data is determined as the energy value of the remote audio data, where the audio value corresponding to the remote audio data is a string of K bits, and K is a positive integer; or, the average value of the audio values ​​corresponding to the remote audio data is determined as the energy value of the remote audio data; or, the median value of the audio values ​​corresponding to the remote audio data is determined as the energy value of the remote audio data. The process of obtaining the energy value of the background audio data includes: The maximum value among the audio values ​​corresponding to the background audio data is determined as the energy value of the background audio data, wherein the audio values ​​corresponding to the background audio data are a string of K bits; or, the average value of the audio values ​​corresponding to the background audio data is determined as the energy value of the background audio data; or, the median value of the audio values ​​corresponding to the background audio data is determined as the energy value of the background audio data.

6. The method according to claim 1, characterized in that, The step of adjusting the energy of the background audio data and / or the energy of the far-end audio data based on the speech detection result, the energy value of the far-end audio data, and the energy value of the background audio data includes: If the speech detection result indicates that the far-end audio data includes a valid speech signal, when the ratio of the energy value of the far-end audio data to the energy value of the background audio data is less than a first preset threshold, the energy of the far-end audio data is increased or the energy of the background audio data is decreased until the ratio of the energy value of the far-end audio data to the energy value of the background audio data after adjustment is greater than the first preset threshold. If the speech detection result indicates that the far-end audio data does not include a valid speech signal, when the ratio of the energy value of the background audio data to the energy value of the far-end audio data is less than a second preset threshold, the energy of the far-end audio data is reduced until the ratio of the energy value of the background audio data to the energy value of the adjusted far-end audio data is greater than the second preset threshold.

7. The method according to claim 1, characterized in that, The step of adjusting the energy of the background audio data and / or the energy of the far-end audio data based on the speech detection result, the energy value of the far-end audio data, and the energy value of the background audio data includes: If the speech detection result indicates that the far-end audio data includes a valid speech signal, when the difference between the energy value of the far-end audio data and the energy value of the background audio data is less than a third preset threshold, the energy of the far-end audio data is increased or the energy of the background audio data is decreased until the difference between the energy value of the adjusted far-end audio data and the energy value of the background audio data is greater than the third preset threshold. If the speech detection result indicates that the far-end audio data does not include a valid speech signal, when the difference between the energy value of the background audio data and the energy value of the far-end audio data is less than a fourth preset threshold, the energy of the far-end audio data is reduced until the difference between the energy value of the background audio data and the energy value of the adjusted far-end audio data is greater than the fourth preset threshold.

8. The method according to claim 1, characterized in that, The acquisition of background audio data and remote audio data in audio and video calls includes: The background audio data can be obtained from the server, or from local storage. Receive N remote voice streams from the server, where N is a positive integer; If N is greater than 1, the N far-end voice streams are mixed to obtain the far-end audio data; if N = 1, the N far-end voice streams are determined as the far-end audio data.

9. The method according to claim 8, characterized in that, The mixing of the N far-end voice streams to obtain the far-end audio data includes: The audio values ​​corresponding to the N remote voice streams are superimposed to obtain the remote audio data.

10. An audio processing apparatus, characterized in that, include: The first acquisition module is used to acquire background audio data and remote audio data in audio and video calls; The voice detection module is used to perform voice detection on the remote audio data and obtain a voice detection result, wherein the voice detection result is whether the remote audio data includes a valid voice signal. The second acquisition module is used to acquire the energy value of the remote audio data; The second acquisition module is further configured to: acquire the energy value of the background audio data; The adjustment module is used to adjust the energy of the background audio data and / or the energy of the far-end audio data based on the speech detection result, the energy value of the far-end audio data, and the energy value of the background audio data, so that the energy values ​​of the adjusted background audio data and the far-end audio data meet preset playback conditions.

11. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store a computer program, the processor being used to invoke and run the computer program stored in the memory to perform the method of any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer program, cause the computer to perform the method as described in any one of claims 1 to 9.

13. A computer program product containing instructions, characterized in that, When the instructions are executed on a computer, the computer performs the method according to any one of claims 1 to 9.

Images