Audio processing method and electronic device

Abstract

The embodiment of the present application discloses an audio processing method and an electronic device, which relates to the field of electronic devices, and can identify the voice of the main speaker and perform sound effect processing on the voice of the main speaker, so that the user can hear the voice of the main speaker clearly and improve the user's experience. The specific scheme is: the first electronic device obtains the call data collected by the second electronic device; the first electronic device determines the audio data of each user among multiple users based on the first audio data; the first electronic device determines the second audio data of the target user and the third audio data of the non-target user based on the audio data of each user; the first electronic device performs sound effect enhancement processing on the second audio data to obtain fourth audio data, and performs sound effect suppression processing on the third audio data to obtain fifth audio data; the first electronic device outputs the target audio data based on the fourth audio data and the fifth audio data.

Description

Technical Field

[0001] The present application relates to the field of electronic devices, and in particular to an audio processing method and electronic device. Background Art

[0002] At present, voice calls have become an important function of electronic devices (such as mobile phones, etc.). For example, a user can use a local device (such as a mobile phone) to make a voice call with a remote device (such as a mobile phone), thereby providing the user with a voice communication function. For example, when a user uses a mobile phone to make a voice call with another mobile phone, the receiver of the user's mobile phone can output the sound signal from the other mobile phone.

[0003] However, when a user uses a local device to make a voice call with a peer device, if the sound signal sent by the peer device includes the voices of multiple speakers, it is easy for the user to not be able to hear the voice of the main speaker, resulting in a poor user experience. Summary of the invention

[0004] The embodiments of the present application provide an audio processing method and an electronic device. When a sound signal sent by a peer device includes the voices of multiple speakers, the method can identify the voice of a main speaker and perform sound effect processing on the voice of the main speaker, so that a user can clearly hear the voice of the main speaker and improve the user's usage experience.

[0005] In order to achieve the above purpose, the embodiment of the present application adopts the following technical solution:

[0006] In a first aspect, an embodiment of the present application provides an audio processing method, which is applied to a first electronic device. The audio processing method may include: the first electronic device obtains call data collected by a second electronic device, the call data includes first audio data, and the first audio data includes audio data of multiple users; the first electronic device determines the audio data of each user among the multiple users based on the first audio data; the first electronic device determines the second audio data of the target user and the third audio data of the non-target user based on the audio data of each user; the non-target user includes users among the multiple users other than the target user; the first electronic device performs sound enhancement processing on the second audio data to obtain fourth audio data, and performs sound suppression processing on the third audio data to obtain fifth audio data; the first electronic device outputs the target audio data based on the fourth audio data and the fifth audio data.

[0007] Based on the method described in the first aspect, the first electronic device can determine the audio data of the target user (i.e., the main speaker) in the audio data, as well as the audio data of non-target users, based on the audio data sent by the second electronic device. That is, the first electronic device can dynamically determine the audio data of the target user in the audio data sent by the second electronic device. Therefore, the first electronic device can determine the audio data of the target user in real time, so that after the main speaker changes, the first electronic device can still enhance and output the audio data of the changed target user, ensuring that the user can hear the audio data of the changed main speaker. Moreover, the solution of the present application can ensure that the user can pay attention to the voices of other speakers while hearing the audio data of the main speaker at the same time because the first electronic device can simultaneously output the audio data of the target user after the enhancement processing and the audio data of the non-target users after the suppression processing.

[0008] In combination with the first aspect, in another possible implementation method, the above-mentioned first electronic device determines the audio data of each user among multiple users based on the first audio data, which may include: the first electronic device performs frame processing on the first audio data to obtain multiple audio frame data; the first electronic device performs sound source separation on the audio frame data to obtain sub-audio data of each user in the audio frame data; the first electronic device obtains the audio data of each user based on the sub-audio data of each user in the multiple audio frame data.

[0009] Based on this possible implementation method, since the first electronic device performs frame processing on the first audio data to obtain multiple audio frame data, the first electronic device can separate the sound sources of the audio frame data to obtain sub-audio data of each user in the audio frame data, and can accurately determine the audio data of each user in the first audio data collected by the second electronic device.

[0010] In combination with the first aspect, in another possible implementation method, the above-mentioned first electronic device obtains the audio data of each user based on the sub-audio data of each user in multiple audio frame data, which may include: the first electronic device extracts the voiceprint feature of the sub-audio data of the first user to obtain the first voiceprint feature; the first user is any one of the multiple users; the first electronic device matches the first voiceprint feature with the second voiceprint feature in the voiceprint library; the second voiceprint feature is the voiceprint feature of the second user, and the second user is any one of the multiple users; when the first voiceprint feature does not match the second voiceprint feature, the first electronic device determines that the first user and the second user are different users, and determines the sub-audio data of the first user as the audio data of the first user; the first electronic device stores the first voiceprint feature in the voiceprint library.

[0011] Based on this possible implementation, since the first electronic device can extract the voiceprint features of the user and match the voiceprint features of the user with the voiceprint features in the voiceprint library, when no match is successful, the first electronic device can determine that the user is not in the voiceprint library, and at the same time, the first electronic device can update the voiceprint library according to the voiceprint features of the user. That is, the first electronic device can determine whether the user is a user already in the voiceprint library according to the voiceprint features of the user and the voiceprint features in the voiceprint library.

[0012] In combination with the first aspect, in another possible implementation, the above-mentioned audio processing method may also include: when the first voiceprint feature matches the second voiceprint feature, the first electronic device determines that the first user and the second user are the same user, and determines that the audio data of the second user includes the sub-audio number of the first user.

[0013] Based on this possible implementation, since the first electronic device can extract the voiceprint features of the user and match the voiceprint features of the user with the voiceprint features in the voiceprint library, when the match is successful, the first electronic device can determine that the user is an existing user in the voiceprint library, and thus can determine that the audio data of the user is the audio data corresponding to the existing user. That is, the first electronic device can determine whether the audio data of the user is the audio data of the same user based on the voiceprint features of the user and the voiceprint features in the voiceprint library.

[0014] In combination with the first aspect, in another possible implementation method, the above-mentioned first electronic device determines the second audio data of the target user and the third audio data of the non-target user based on the audio data of each user, which may include: the first electronic device determines the speaking time and sound energy of each user based on the audio data of each user; the first electronic device determines the target user based on the speaking time and sound energy of each user; the target user is the user with the longest speaking time and / or the largest sound energy among multiple users; the first electronic device determines the audio data of the target user as the second audio data, and determines the audio data of the non-target user as the third audio data.

[0015] Based on this possible implementation, the first electronic device can determine the speaking time and sound energy of each user, and determine the user with the longest speaking time and / or the largest sound energy among multiple users as the target user, and can accurately identify that the first audio data includes the audio data of the main speaker among the audio data of multiple users. Thus, the first electronic device can perform sound effect enhancement processing on the audio data of the target user, so that when the first electronic device outputs the audio data of the target user, the user of the first electronic device can hear the voice of the target user clearly.

[0016] In combination with the first aspect, in another possible implementation method, the above-mentioned call data may also include first image data corresponding to the first audio data, and the above-mentioned first electronic device determines the second audio data of the target user and the third audio data of the non-target user according to the audio data of each user, which may include: the first electronic device determines the lip movement information of each user in the first audio data according to the first image data; the first electronic device determines the probability of each user speaking according to the lip movement information of each user and the audio data of each user; the first electronic device determines the target user among multiple users according to the probability of each user speaking; the first electronic device determines the audio data of the target user as the second audio data, and determines the audio data of the non-target user as the third audio data.

[0017] Based on this possible implementation, when determining the target user, the first electronic device can determine the lip movement information of each user based on the first image data corresponding to the first audio data, and can accurately determine the speaking probability of each user in combination with the lip movement information of each user. The user with the highest speaking probability among the speaking probabilities of each user is determined as the target user, and the target user can be accurately determined. Thus, the first electronic device can perform sound enhancement processing on the audio data of the target user, so that when the first electronic device outputs the audio data of the target user, the user of the first electronic device can hear the voice of the target user clearly.

[0018] In combination with the first aspect, in another possible implementation, the above-mentioned first electronic device performs sound enhancement processing on the second audio data to obtain fourth audio data, and performs sound suppression processing on the third audio data to obtain fifth audio data, which may include: the first electronic device performs sound enhancement processing on the second audio data according to the transmission function corresponding to the first direction angle and / or the first gain attenuation to obtain the fourth audio data; the first electronic device performs sound suppression processing on the third audio data according to the transmission function corresponding to the second direction angle and / or the second gain attenuation to obtain the fifth audio data; wherein the first direction angle is smaller than the second direction angle, and / or the first gain attenuation is greater than the second gain attenuation.

[0019] Based on this possible implementation, since the first electronic device can perform sound enhancement processing on the second audio data according to the transmission function corresponding to the first direction angle, and perform sound suppression processing on the third audio data according to the transmission function corresponding to the second direction angle, and the first direction angle is smaller than the second direction angle, the audio data of the target user and the audio data of the non-target user can be separated from the spatial position. Thus, the user of the first electronic device can hear the voice of the target user clearly and pay attention to the voice of the non-target user. Since the first electronic device can perform sound enhancement processing on the second audio data according to the first gain attenuation, and perform sound suppression processing on the third audio data according to the second gain attenuation, and the first gain attenuation coefficient is greater than the second gain attenuation, it can be ensured that the amplitude change of the audio data of the target user during the transmission process is less than the amplitude change of the audio data of the non-target user during the transmission process, so that the user of the first electronic device can hear the voice of the target user clearly and pay attention to the voice of the non-target user.

[0020] In combination with the first aspect, in another possible implementation, the non-target users may include a third user and a fourth user, the third audio data may include audio data of the third user and audio data of the fourth user, the second direction angle may include a first sub-direction angle and a second sub-direction angle, the first sub-direction angle may be smaller than the second sub-direction angle, and the first electronic device may perform sound effect suppression processing on the third audio data according to a transmission function corresponding to the second direction angle to obtain fifth audio data, which may include: the first electronic device may perform sound effect suppression processing on the audio data of the third user according to the transmission function corresponding to the first sub-direction angle to obtain sixth audio data; the first electronic device may perform sound effect suppression processing on the audio data of the fourth user according to the transmission function corresponding to the second sub-direction angle to obtain seventh audio data; wherein the fifth audio data includes the sixth audio data and the seventh audio data.

[0021] Based on this possible implementation, since the first electronic device can suppress the audio data of different users among the non-target users according to the transmission function corresponding to the first sub-direction angle and the transmission function corresponding to the second sub-direction angle, the audio data of different users among the non-target users can be further separated in spatial positions. This can further enable the user of the first electronic device to hear the voice of the target user clearly and also pay attention to the voices of different non-target users.

[0022] In combination with the first aspect, in another possible implementation, the non-target users may include a fifth user and a sixth user, the third audio data may include audio data of the fifth user and audio data of the sixth user, the second gain attenuation may include a first sub-gain attenuation and a second sub-gain attenuation, the first gain attenuation is greater than the second gain attenuation, and the first electronic device performs sound effect suppression processing on the third audio data according to the second gain attenuation to obtain fifth audio data, which may include: the first electronic device performs sound effect suppression processing on the audio data of the fifth user according to the first sub-gain attenuation to obtain eighth audio data; the first electronic device performs sound effect suppression processing on the audio data of the sixth user according to the second sub-gain attenuation to obtain ninth audio data; wherein the fifth audio data includes the eighth audio data and the ninth audio data.

[0023] Based on this possible implementation method, since the first electronic device can suppress the audio data of different users among non-target users according to the first sub-gain attenuation and the second sub-gain attenuation, it can further ensure that the audio data of different users among non-target users have different amplitude changes during the transmission process, thereby enabling the user of the first electronic device to clearly hear the voice of the target user and also pay attention to the voices of different users among non-target users.

[0024] In combination with the first aspect, in another possible implementation, the first electronic device outputs target audio data based on the fourth audio data and the fifth audio data, which may include: the first electronic device accumulates the fourth audio data and the fifth audio data to obtain the target audio data; and the first electronic device outputs the target audio data.

[0025] Based on this possible implementation method, since the first electronic device can accumulate the fourth audio data and the fifth audio data to obtain the target audio data, when outputting the target audio data, the user of the first electronic device can clearly hear the voice of the target user and can also pay attention to the voices of different users among non-target users.

[0026] In a second aspect, an embodiment of the present application provides an audio processing device, which can be applied to a first electronic device to implement the method in the first aspect. The functions of the audio processing device can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, for example, an acquisition module, a determination module, a processing module, and an output module.

[0027] The acquisition module may be used to acquire call data collected by the second electronic device, where the call data includes first audio data, and the first audio data includes audio data of multiple users.

[0028] The determination module can be used by the first electronic device to determine the audio data of each user among the multiple users based on the first audio data.

[0029] The determination module may also be used to determine the second audio data of the target user and the third audio data of the non-target user according to the audio data of each user; the non-target user includes users other than the target user among the multiple users.

[0030] The processing module can be used to perform sound effect enhancement processing on the second audio data to obtain fourth audio data, and to perform sound effect suppression processing on the third audio data to obtain fifth audio data.

[0031] The output module can be used to output target audio data based on the fourth audio data and the fifth audio data.

[0032] In a third aspect, an audio processing device is provided, which has the function of implementing the method described in the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0033] In a fourth aspect, an audio processing device is provided, comprising: a processor and a memory; the memory is used to store computer execution instructions, and when the audio processing device is running, the processor executes the computer execution instructions stored in the memory to enable the audio processing device to perform the audio processing method as described in any one of the first aspects above.

[0034] In a fifth aspect, an audio processing device is provided, comprising: a processor; the processor is used to couple with a memory, and after reading instructions in the memory, execute the audio processing method as described in any one of the above-mentioned first aspects according to the instructions.

[0035] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by an electronic device, the electronic device implements the audio processing method as described in any one of the first aspect or possible implementations of the first aspect.

[0036] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising a computer-readable code, which, when executed in an electronic device, enables the electronic device to implement an audio processing method as described in any one of the first aspect or possible implementations of the first aspect.

[0037] In an eighth aspect, a device (for example, the device may be a chip system) is provided, the device including a processor for supporting an electronic device to implement the functions involved in the first aspect above. In one possible design, the device also includes a memory for storing program instructions and data necessary for the electronic device. When the device is a chip system, it may be composed of a chip, or may include a chip and other discrete devices.

[0038] It should be understood that the beneficial effects of the second to eighth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of a call scenario in the related art;

[0040] Figure 2 A schematic diagram of a call scenario provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of an audio processing method provided in an embodiment of the present application Figure 1 ;

[0044] Figure 6 A schematic diagram of a flow chart of a sound source separation method provided in an embodiment of the present application;

[0045] Figure 7 A schematic diagram of a process for determining audio data of different speakers provided in an embodiment of the present application;

[0046] Figure 8 Schematic diagram of the process of determining the main speaker provided in the embodiment of the present application Figure 1 ;

[0047] Fig. 9 Schematic diagram of the process of determining the main speaker provided in the embodiment of the present application Figure 2 ;

[0048] Fig.10 A schematic diagram of an audio processing method provided in an embodiment of the present application Figure 2 ;

[0049] Fig.11 A schematic diagram of the structure of an audio processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0052] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting of the present application. As used in the specification and appended claims of the present application, the singular expressions "a", "an", "said", "above", "the" and "this" are intended to also include plural expressions, unless the context clearly indicates otherwise. It should also be understood that " / " means or, for example, A / B can mean A or B; "and / or" in the text is merely a description of the association relationship of associated characters, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0053] At present, voice calls have become an important function of electronic devices (such as mobile phones, etc.). For example, a user can use a local device (such as a mobile phone) to make a voice call with a remote device (such as another mobile phone, etc.), thereby providing the user with a voice communication function. For example, when a user uses a mobile phone to make a voice call with another mobile phone, the receiver of the user's mobile phone can output the sound signal from the other mobile phone.

[0054] However, when a user uses a local device to make a voice call with a peer device (such as another mobile phone), if the sound signal sent by the peer device includes the voices of multiple speakers, or the sound signal of the peer device includes background noise, it is easy for the user to not be able to hear the voice of the main speaker, resulting in a poor user experience.

[0055] For example, combined with Figure 1As shown, when user A uses mobile phone 01 to make a voice call with user B, mobile phone 01 (i.e., the local device) can receive the sound signal sent by mobile phone 02 (i.e., the opposite device) of user B, so that mobile phone 01 can output the sound signal sent by mobile phone 02. The sound signal sent by mobile phone 02 includes the voice of user B, that is, user B is the main speaker, and can also be called the target speaker.

[0056] When the sound signal sent by mobile phone 02 to mobile phone 01 is the voice of user B, if user C and user D are also speaking, the sound signal sent by mobile phone 02 to mobile phone 01 includes not only the voice of user B, but also the voices of user C and user D.

[0057] When mobile phone 01 outputs the sound signal sent by mobile phone 02 (including the voice of user B, and also the voices of user C and user D), user A may not be able to hear the voice of user B clearly, resulting in a poor user experience.

[0058] In some scenarios, when the sound signal sent by mobile phone 02 to mobile phone 01 is the voice of user B, if the environmental noise (i.e., background noise) where user B is located is large, the sound signal sent by mobile phone 02 to mobile phone 01 includes not only the voice of user B but also the background noise. When mobile phone 01 outputs the sound signal sent by mobile phone 02 (including the voice of user B and the background noise), user A may not be able to hear the voice of user B clearly, resulting in a poor user experience.

[0059] That is to say, in a call scenario, if the sound signal sent by the other device includes the voices of multiple speakers, or the sound signal sent by the other device includes background noise, when the local device outputs the sound signal sent by the other device including the voices of multiple speakers, or when the local device outputs the sound signal sent by the other device including background noise, it is easy for the user of the local device to be unable to hear the voice of the main speaker, resulting in a poor user experience.

[0060] In order to solve the problem that the sound signal sent by the other end device in a call scenario includes the voices of multiple speakers, or the sound signal sent by the other end device includes background noise, resulting in the user of the local device being unable to hear the voice of the main speaker, resulting in a poor user experience, the relevant technology generally samples the following two methods to process the sound signal sent by the other end device.

[0061] The first method is to eliminate background noise. That is, the local device can eliminate the background noise of the sound signal sent by the opposite device, so that when the local device outputs the sound signal sent by the opposite device, the background noise can be avoided from affecting the voice of the main speaker. However, this method cannot eliminate the interference of other speakers' voices on the main speaker's voice. That is, although this method can eliminate the background noise included in the sound signal sent by the opposite device, it cannot remove the interfering voice, resulting in that when the sound signal sent by the opposite device includes the voices of multiple speakers, the user of the local device still cannot hear the main speaker's voice clearly.

[0062] The second method is to retain the voice of the target speaker and eliminate the voices of other speakers. That is, the local device first registers the voice of the target speaker (that is, the voice signal of the target speaker). When the local device receives the sound signal sent by the opposite device including the voices of multiple speakers, the local device can extract the voice of the target speaker and eliminate the voices of other speakers. Therefore, when the local device outputs the sound signal sent by the opposite device, it can avoid the influence of the voices of other speakers on the voice of the main speaker. However, this method directly eliminates the voices of other speakers, resulting in the local user being unable to hear the voice of the main speaker (that is, the target speaker) while also being able to hear the voices of other speakers. That is, this method cannot enable the local user to pay attention to the voices of other speakers while hearing the voice of the main speaker (that is, the target speaker).

[0063] In addition, the second method mentioned above requires the target speaker's voice to be registered on the local device first, which results in the inability to dynamically extract the target speaker's voice. Therefore, when the target speaker changes, but the local device has not registered the changed target speaker's voice, the local device cannot extract the changed target speaker's voice and eliminate the voices of other people. Therefore, when the local device outputs the sound signal sent by the other device, it still cannot avoid the influence of other speakers' voices on the main speaker's (i.e., the changed target speaker) voice.

[0064] Through the above analysis of the first and second methods, it can be seen that the relevant technology has not well solved the problem that in a call scenario, the sound signal sent by the other end device includes the voices of multiple speakers, or the sound signal sent by the other end device includes background noise, resulting in the user of the local device being unable to hear the voice of the main speaker, resulting in a poor user experience.

[0065] In response to the above problems, an embodiment of the present application provides an audio processing method, which is applied to a first electronic device. After the first electronic device (i.e., a local device, such as a mobile phone) can receive audio data (i.e., a sound signal) sent by a second electronic device (i.e., a peer device, such as another mobile phone), the first electronic device can dynamically determine the audio data of the main speaker in the audio data, as well as the audio data of other speakers and the audio data of noise. Afterwards, the first electronic device can enhance the audio data of the main speaker and suppress the audio data of other speakers and the audio data of noise. Afterwards, the first electronic device can simultaneously output the audio data of the main speaker after the enhancement processing and the audio data of other speakers and the audio data of noise after the suppression processing.

[0066] The solution of the present application is that the first electronic device can dynamically determine the audio data of the main speaker in the audio data, as well as the audio data of other speakers and the audio data of noise. Therefore, the first electronic device can determine the audio data of the main speaker in real time, so that after the main speaker changes, the first electronic device still enhances and outputs the audio data of the changed main speaker, ensuring that the user can clearly hear the audio data of the changed main speaker. Moreover, the solution of the present application is that the first electronic device can simultaneously output the audio data of the main speaker after enhancement processing and the audio data of other speakers and the audio data of noise after suppression processing, ensuring that the user can pay attention to the voices of other speakers while hearing the audio data of the main speaker clearly.

[0067] For example, combined with Figure 2 As shown, when user A uses mobile phone 01 (i.e., the local device, the first electronic device) to make a voice call with user B, mobile phone 01 can receive the sound signal (i.e., audio data) sent by mobile phone 02 (i.e., the opposite device, the second electronic device) of user B, so that mobile phone 01 can output the sound signal sent by mobile phone 02. The sound signal sent by mobile phone 02 includes the voice of user B, that is, user B is the main speaker, and can also be called the target speaker.

[0068] When the sound signal sent by mobile phone 02 to mobile phone 01 is the voice of user B, if user C and user D are also speaking, the sound signal sent by mobile phone 02 to mobile phone 01 includes not only the voice of user B, but also the voices of user C and user D.

[0069] When mobile phone 01 outputs the sound signal sent by mobile phone 02 (including the voice of user B, and also the voices of user C and user D), mobile phone 01 can dynamically determine the voice of the main speaker (i.e. user B) and the voices and noise of user C and user D in the sound signal sent by mobile phone 02. Afterwards, mobile phone 01 can enhance the voice of user B (i.e. the audio data of user B) and suppress the voices and noise of user C and user D. Afterwards, mobile phone 01 can simultaneously output the voice of user B after enhancement and the voices and noise of user C and user D after suppression, which can ensure that the user can pay attention to the voices of other speakers (i.e. user C and user D) while hearing the voice of the main speaker (i.e. user B) clearly.

[0070] In addition, when the main speaker is user B or user C, mobile phone 01 can dynamically determine the voice of the main speaker (i.e. user C) in the sound signal sent by mobile phone 02, as well as the voices and noises of user B and user D. Afterwards, mobile phone 01 can enhance the voice of user C (i.e. the audio data of user C) and suppress the voices and noises of user B and user D. Afterwards, mobile phone 01 can simultaneously output the voice of user C after enhancement and the voices and noises of user B and user D after suppression, which can ensure that the user can pay attention to the voices of other speakers (i.e. user B and user D) while hearing the voice of the main speaker (i.e. user C) clearly. That is, after the main speaker changes, mobile phone 01 can still enhance and output the voice of the changed main speaker (i.e. user C), ensuring that the user can hear the voice of the changed main speaker clearly.

[0071] The audio processing method provided in the embodiment of the present application is described below.

[0072] The audio processing method provided in the embodiment of the present application can be applied to the above-mentioned first electronic device. In some embodiments, the above-mentioned first electronic device can be a mobile phone, a tablet computer, a handheld computer, a personal computer (PC), a cellular phone, a personal digital assistant (PDA), and other electronic devices capable of making voice calls. The embodiment of the present application does not limit the specific form of the electronic device.

[0073] The second electronic device may be a mobile phone, tablet computer, handheld computer, PC, cellular phone, PDA or other electronic device capable of making voice calls. The specific form of the electronic device is not limited in the present embodiment. It should be noted that the specific form of the second electronic device may be the same as the first electronic device or different from the first electronic device, and the present embodiment does not limit this.

[0074] For example, taking the first electronic device as a mobile phone, Figure 3 A schematic structural diagram of a first electronic device provided in an embodiment of the present application is shown.

[0075] like Figure 3 As shown, the first electronic device may include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, an earphone interface 370D, a sensor module 380, a button 390, a motor 391, an indicator 392, a camera 393, a display screen 394, and a subscriber identification module (SIM) card interface 395, etc. Among them, the sensor module 380 may include a pressure sensor 380A, a gyroscope sensor 380B, an air pressure sensor 380C, a magnetic sensor 380D, an acceleration sensor 380E, a distance sensor 380F, a proximity light sensor 380G, a fingerprint sensor 380H, a temperature sensor 380J, a touch sensor 380K, an ambient light sensor 380L, a bone conduction sensor 380M, etc.

[0076] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the first electronic device. In other embodiments, the first electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0077] The processor 310 may include one or more processing units, for example, the processor 310 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0078] The controller can be the nerve center and command center of the first electronic device. The controller can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0079] The processor 310 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 310 is a cache memory. The memory may store instructions or data that the processor 310 has just used or cyclically used. If the processor 310 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 310, and thus improves the efficiency of the system.

[0080] In some embodiments, the processor 310 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0081] The wireless communication function of the first electronic device can be implemented through antenna 1, antenna 2, mobile communication module 350, wireless communication module 360, a modem processor, and a baseband processor.

[0082] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the first electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0083] The mobile communication module 350 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the first electronic device. The mobile communication module 350 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 350 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 350 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 350 can be set in the processor 310. In some embodiments, at least some of the functional modules of the mobile communication module 350 can be set in the same device as at least some of the modules of the processor 310.

[0084] The wireless communication module 360 ​​can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the first electronic device. The wireless communication module 360 ​​can be one or more devices integrating at least one communication processing module. The wireless communication module 360 ​​receives electromagnetic waves via antenna 2, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 310. The wireless communication module 260 can also receive the signal to be sent from the processor 310, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through antenna 2.

[0085] In some embodiments, the antenna 1 of the first electronic device is coupled to the mobile communication module 350, and the antenna 2 is coupled to the wireless communication module 360, so that the first electronic device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology.

[0086] The first electronic device implements the display function through a GPU, a display screen 394, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 394 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 310 may include one or more GPUs that execute program instructions to generate or change display information.

[0087] The display screen 394 is used to display images, videos, etc. The display screen 394 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the first electronic device may include 1 or N display screens 394, where N is a positive integer greater than 1.

[0088] The first electronic device can implement a shooting function through an ISP, a camera 393, a video codec, a GPU, a display screen 394, and an application processor. In some embodiments, the first electronic device may include 1 or N cameras 393, where N is a positive integer greater than 1.

[0089] The internal memory 321 can be used to store computer executable program codes, which include instructions. The processor 310 executes various functional applications and data processing of the first electronic device by running the instructions stored in the internal memory 321. The internal memory 321 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data established during the use of the first electronic device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 321 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0090] The acceleration sensor 380E can periodically collect acceleration data of the first electronic device at a certain frequency. For example, the acceleration magnitude of the first electronic device in various directions (generally XYZ three-axis directions) can be collected.

[0091] Of course, it is understandable that the above Figure 3 The figure is only an exemplary description when the first electronic device is in the form of a mobile phone. If the first electronic device is in the form of a tablet computer, a handheld computer, a PC, a PDA, a wearable device (such as a smart watch, a smart bracelet), etc., the structure of the first electronic device may include Figure 3 The structure shown in the figure is less than Figure 3 More structures are shown in the figure, which are not limited here.

[0092] It is understandable that, generally speaking, the realization of the function of the first electronic device requires not only hardware support but also software cooperation.

[0093] The software system of the first electronic device may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. Taking the system as an example, the software structure of the first electronic device is exemplified.

[0094] like Figure 4 As shown, the system used by electronic equipment is Taking the system as an example, in the embodiment of the present application, the software of the electronic device is divided into four layers, namely, the application layer, the application framework layer, the system library and the Runtime( runtime), HAL layer (hardware abstraction layer) and driver layer (also called kernel layer).

[0095] The application layer can include a series of applications. Figure 4 As shown, the application layer may include applications such as camera, gallery, calendar, map, WLAN, Bluetooth, music, video, short message, call, etc.

[0096] Among them, the application layer can also include a behavioral audio processing application. The audio processing application can be used to execute the following audio processing method. That is, the audio processing application can obtain the audio data sent by the second electronic device, and dynamically determine the audio data of the main speaker in the audio data, as well as the audio data of other speakers and the audio data of noise. Afterwards, the audio processing application can be used to enhance the audio data of the main speaker and suppress the audio data of other speakers and the audio data of noise. Afterwards, the audio processing application can simultaneously output the audio data of the main speaker after enhancement processing and the audio data of other speakers and the audio data of noise after suppression processing through the earpiece of the first electronic device.

[0097] The application framework layer can provide an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0098] like Figure 4 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0099] In an embodiment of the present application, if there is no audio processing application in the application layer, an audio processing service having the functions of the audio processing application may exist in the application framework layer of the first electronic device.

[0100] like Figure 4 As shown, Runtime The runtime includes the core library and the virtual machine. Runtime The runtime is responsible for Scheduling and management of the system.

[0101] The core library consists of two parts: one is the function that the Java language needs to call, and the other is The core library.

[0102] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0103] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0104] The HAL layer is an interface layer between the operating system kernel and the hardware circuit, and its purpose is to abstract the hardware. It hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system, making it hardware-independent and portable on multiple platforms. The HAL layer provides a standard interface to display device hardware functions to the higher-level Java API framework (i.e., the framework layer). The HAL layer contains multiple library modules, each of which implements an interface for a specific type of hardware component, such as: audio HAL audio module, bluetooth HAL Bluetooth module, camera HAL camera module, sensors HAL sensor module (or Isensor service, sensor service).

[0105] like Figure 4 As shown, in an embodiment of the present application, if there is no audio processing application in the application program layer, the audio module in the HAL layer of the first electronic device may have the function of the audio processing application.

[0106] The kernel layer is a layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, sensor driver, etc., which is not limited in the embodiments of the present application. Among them, the sensor driver can specifically include the driver of each sensor included in the electronic device, such as a nine-axis sensor driver, which can include a three-axis gyroscope sensor driver, a three-axis magnetic sensor driver, and a three-axis acceleration sensor driver. The audio driver can be used to drive the earpiece of the first electronic device to simultaneously output the audio data of the main speaker after enhancement processing and the audio data of other speakers and the audio data of noise after suppression processing.

[0107] The following is combined with Figure 5 , the audio processing method provided in the embodiment of the present application is described in detail. Figure 5 As shown, the audio processing method may include the following S501-S505.

[0108] S501: A first electronic device receives audio data sent by a second electronic device.

[0109] In some examples, after the first electronic device establishes a voice call with the second electronic device, the first electronic device may receive audio data (also referred to as first audio data in the embodiment of the present application) sent by the second electronic device. The audio data sent by the second electronic device may include audio data of multiple speakers (also referred to as multiple users in the embodiment of the present application), and the audio data sent by the second electronic device may also include audio data of noise, such as the sound of a device such as a television when the speaker of the second electronic device is speaking.

[0110] The first electronic device and the second electronic device establish a voice call, which can be established by the first electronic device and the second electronic device through an application program provided by the first electronic device, or by a third-party application program. This is not limited in the embodiments of the present application.

[0111] In other examples, after the first electronic device establishes a video call with the second electronic device, the first electronic device may also receive audio data sent by the second electronic device.

[0112] It should be noted that after the first electronic device establishes a video call with the second electronic device, when the first electronic device receives audio data sent by the second electronic device, the first electronic device may also receive image data (which may also be referred to as first image data in the embodiment of the present application) sent by the second electronic device.

[0113] In some examples, the audio data sent by the second electronic device may be single-microphone data or multi-microphone data. This is not limited in the embodiments of the present application. Single-microphone data may be audio data collected by the second electronic device using a single microphone. Single-microphone data may include a mono audio signal (i.e., mono audio data) collected from a sound source or environment. Multi-microphone data may be audio data collected simultaneously by the second electronic device using multiple microphones. Multi-microphone data may include multi-channel audio signals (i.e., multi-channel audio data) from multiple sound sources collected by multiple microphones.

[0114] In some examples, the first electronic device receives audio data sent by the second electronic device, and the HAL layer of the first electronic device (such as the audio module included in the HAL layer) can receive the audio data (i.e., single microphone data or multi-microphone data) sent by the second electronic device. Thus, the HAL layer (such as the audio module included in the HAL layer) can determine the audio data of each speaker based on the audio data sent by the second electronic device. That is, the HAL layer (such as the audio module included in the HAL layer) can perform sound source separation on the audio data sent by the second electronic device.

[0115] In some examples, the first electronic device receives audio data sent by the second electronic device, which may be audio data sent by the second electronic device and received in real time by the first electronic device, such as when the first electronic device and the second electronic device are in a voice call, the first electronic device receives audio data sent by the second electronic device in real time. The first electronic device receives audio data sent by the second electronic device, which may be audio data sent by the second electronic device and received and stored by the first electronic device.

[0116] It should be noted that, in the embodiment of the present application, the first electronic device receives the audio data sent by the second electronic device, and executes the solution of the present application on the audio data sent by the second electronic device. The first electronic device may also execute the solution of the present application on the audio data stored in the first electronic device.

[0117] S502: The first electronic device determines audio data of each speaker according to the audio data sent by the second electronic device.

[0118] After the first electronic device receives the audio data sent by the second electronic device, the first electronic device can determine the audio data of each speaker based on the audio data sent by the second electronic device. That is, the first electronic device can perform sound source separation on the audio data sent by the second electronic device, thereby determining the audio data of each speaker in the audio data sent by the second electronic device (which can also be referred to as the audio data of each user in the embodiment of the present application).

[0119] In some examples, when the first electronic device performs sound source separation on the audio data sent by the second electronic device and determines the audio data of each speaker in the audio data sent by the second electronic device, the first electronic device can also determine the audio data of the noise in the audio data sent by the second electronic device.

[0120] In some examples, the first electronic device performs sound source separation on the audio data sent by the second electronic device. The first electronic device may perform sound source separation on the audio data sent by the second electronic device through a multi-sound source separation algorithm, thereby separating the audio data of each speaker and the audio data of noise in the audio data sent by the second electronic device.

[0121] It should be noted that when the first electronic device performs sound source separation on the audio data sent by the second electronic device and determines the audio data of each speaker and the audio data of noise in the audio data sent by the second electronic device, the time information corresponding to the audio data of each speaker and the time information corresponding to the audio data of noise can also be determined. The time information corresponding to the audio data of each speaker is the time point at which different speakers speak in the audio data sent by the second electronic device. The time information corresponding to the audio data of noise is the time point at which there is noise in the audio data sent by the second electronic device.

[0122] The multi-source separation algorithm can be used to separate individual sound sources from mixed audio data (ie, mixed signals), ie, audio data of each speaker and audio data of noise.

[0123] In some examples, algorithms for separating multiple sound sources may include methods based on blind source separation, deep learning models, spectral subtraction, independent low-rank matrix decomposition, encoder-decoder structures, and the like.

[0124] The blind source separation (BSS) method can use statistical principles to decompose the mixed signal through independent component analysis (ICA), non-negative matrix factorization (NMF) and other techniques to estimate and separate the various sound sources in the original signal.

[0125] The deep learning model method can separate mixed signals through the features learned through training. Deep learning models can include convolutional neural networks (CNN) and recurrent neural networks (RNN).

[0126] Spectral subtraction separates mixed signals by estimating and subtracting the spectral information of noise.

[0127] Independent low-rank matrix factorization: This is a method based on the theory of low-rank matrix decomposition, which assumes that the mixed signal can be decomposed into the sum of a matrix with low-rank properties and a sparsity matrix. By decomposing and constraining the mixed signal, the matrix of the original sound source can be estimated.

[0128] The encoder-decoder structure is a common neural network structure, that is, the encoder-decoder network structure is used to process sequence data, especially widely used in natural language processing (NLP) tasks.

[0129] In the encoder-decoder structure, the encoder is responsible for converting the input sequence into a fixed-length vector representation, while the decoder uses the vector representation to generate the output sequence. The whole process can be regarded as a conversion process that maps the input sequence to the output sequence.

[0130] Specifically, the encoder usually uses a recurrent neural network (RNN) or a self-attention mechanism (such as Transformer) to process the input sequence, and gradually obtains a global vector representation, also called a context vector or encoding vector, by gradually processing each element in the sequence. This vector contains the information of the input sequence and is passed to the decoder.

[0131] The decoder is another RNN or Transformer network, which takes the encoding vector as input and generates each element of the output sequence step by step, thereby achieving tasks such as target sequence generation or sequence labeling. In each step of the decoder generation, it comprehensively considers the previously generated content and the information of the encoding vector.

[0132] In the embodiments of the present application, the specific type of the algorithm for multiple sound source separation is not limited, that is, it can separate each individual sound source (such as audio data of different speakers and audio data of noise) from the mixed audio data (i.e., mixed signal). In the embodiments of the present application, the algorithm for multiple sound source separation is used as an example of a method based on an encoder-decoder structure.

[0133] The following is combined with Figure 6 Taking the algorithm of multiple sound source separation as a method based on the encoder-decoder structure, the audio data sent by the second electronic device includes the audio data of multiple speakers and the audio data of noise as an example, an example is given of the first electronic device in an embodiment of the present application determining the audio data of each speaker based on the audio data sent by the second electronic device.

[0134] After the first electronic device receives the audio data sent by the second electronic device, the first electronic device can perform frame processing on the audio data sent by the second electronic device. That is, the first electronic device can divide the audio data sent by the second electronic device into multiple frame data (also referred to as multiple audio frame data in the embodiment of the present application) according to the preset duration.

[0135] It should be noted that the preset duration can be set according to actual conditions, and the specific value of the preset duration is not limited in the embodiment of the present application. For example, the preset duration can be 5 milliseconds.

[0136] After the first electronic device can divide the audio data sent by the second electronic device into multiple frame data according to the preset time length, for each frame data, the first electronic device can separate the sound source thereon, thereby obtaining the audio data of each speaker and the audio data of noise corresponding to each frame data, which may also be referred to as sub-audio data of each user in the audio frame data in the embodiment of the present application.

[0137] Each frame of data can also be called a (Microphone, MIC) signal. Figure 6 As shown, after the first electronic device obtains each frame data, the frame data is first upgraded through the encoder. The encoder can use a multi-layer CNN network, and the encoder can encode the frame data from the frame data in the low-dimensional space to the frame data in the high-dimensional separable space. Since the frame data in the low-dimensional space (i.e., the low-dimensional signal) cannot distinguish the audio data of different speakers and the audio data of noise in the frame data, for example, it is impossible to distinguish from a section of audio which speakers account for each section of the audio. The audio data of different speakers and the audio data of noise included in the frame data can be distinguished by the frame data in the high-dimensional separable space.

[0138] Low-dimensional space refers to a mathematical space with fewer dimensions. In low-dimensional space, data points can be described and represented with fewer coordinates or features. In contrast, high-dimensional separable space requires more coordinates or features to fully describe the data.

[0139] A high-dimensional separable space refers to a specific partitioning method in a high-dimensional feature space, which enables data points to be effectively divided into different categories or clusters by this partitioning method. In the field of machine learning and data mining, data is usually represented as vectors and these vectors are placed in the feature space. The dimension of the feature space is determined by the number of features of each data point, where each feature is represented by one dimension. When there are few features, the feature space is low-dimensional; when there are many features, the feature space is high-dimensional.

[0140] For example, when the features of the frame data include time, the dimension of the frame data is one dimension, and the feature space of the frame data can be a low-dimensional space. When the features of the frame data include multiple features such as time and frequency, the dimension of the frame data is multiple dimensions, and the feature space of the frame data can be a high-dimensional separable space.

[0141] like Figure 6As shown, after the first electronic device upgrades the frame data through the encoder to obtain frame data in a high-dimensional separable space, the first electronic device can pass the frame data in the high-dimensional separable space through a separation network (conformer) to separate the sound sources, thereby obtaining audio data in the high-dimensional separable space of different speakers in the frame data and audio data in the high-dimensional separable space of noise.

[0142] The conformer is a neural network model based on the Transformer structure, mainly used for speech and audio processing tasks. It combines the self-attention mechanism in the Transformer with the convolutional neural network (CNN) to improve the modeling ability on time series data.

[0143] In the speech separation task, the separation network (conformer) can be used to separate multiple sound sources from mixed audio signals. Its design idea is to capture the long-range dependencies in the input sequence through the self-attention mechanism, and use 1D convolution to process the local temporal structure.

[0144] The main components of the separation network (conformer) can include: multi-head self-attention module, causal and non-causal normalization layer, feed-forward neural network, causal and non-causal normalization layer, etc.

[0145] The multi-layer self-attention module can be used to interact and associate elements at different positions in the input sequence to capture global contextual information. The deep residual convolution module can use 1D convolution to extract local features of the sequence to capture the local temporal structure. The feedforward neural network can be used to perform nonlinear transformation and mapping on the features at each position. The context normalization layer can be used to standardize and normalize the features to improve the robustness and convergence of the model. The separation network (conformer) can better handle long-term sequence modeling and has achieved good performance in tasks such as speech separation.

[0146] Before passing the frame data of the high-dimensional separable space through the separation network (conformer) for sound source separation, the separation network (conformer) can be trained with sample data (i.e., the frame data of the high-dimensional separable space and the audio data of the high-dimensional separable space of different speakers in the frame data and the audio data of the high-dimensional separable space of noise). Thus, the first electronic device can use the trained separation network (conformer) to pass the frame data of the high-dimensional separable space for sound source separation, thereby obtaining the audio data of the high-dimensional separable space of different speakers in the frame data and the audio data of the high-dimensional separable space of noise.

[0147] It should be noted that when the frame data of the high-dimensional separable space is passed through a separation network (conformer) for sound source separation, the speech timing of the previous and subsequent segments can also be modeled through the separation network (conformer), so that the previous historical information can be used to assist the current frame in sound source separation. In other words, when the frame data of the high-dimensional separable space is passed through a separation network (conformer) for sound source separation, the information of the previous frame data can be used to model the speech timing, so that when the sound source is separated for the current frame data, the audio data of the high-dimensional separable space of different speakers and the audio data of the high-dimensional separable space of noise in the current frame data can be more accurately determined.

[0148] like Figure 6 As shown, after the first electronic device obtains the audio data of the high-dimensional separable space of different speakers and the audio data of the high-dimensional separable space of noise in the frame data, the audio data of the high-dimensional separable space of different speakers and the audio data of the high-dimensional separable space of noise can be reduced in dimension by a decoder to obtain the audio data of the low-dimensional space of different speakers and the audio data of the low-dimensional space of noise.

[0149] The decoder can also use a multi-layer CNN network, and the decoder can reduce the dimension of the frame data from the high-dimensional separable space to the low-dimensional space. For example, the audio data of the high-dimensional separable space of different speakers and the audio data of the high-dimensional separable space of noise can be converted back to the time domain by the decoder network, that is, the audio data of the high-dimensional separable space of different speakers and the audio data of the high-dimensional separable space of noise can be reduced to the corresponding waveform file.

[0150] like Figure 6As shown, after the first electronic device obtains the audio data of the low-dimensional space of different speakers and the audio data of the low-dimensional space of noise, the first electronic device can pass the audio data of the low-dimensional space of different speakers and the audio data of the low-dimensional space of noise through a fully connected layer (FC) to obtain the masking value of each sound source. That is, the masking value of the audio data of each speaker and the masking value of the audio data of noise can be obtained through the fully connected layer (FC).

[0151] A fully connected layer (FC) connects all neurons in the previous layer to all neurons in the current layer, and each connection has a weight. A fully connected layer maps input data to output data, capturing features in the input data by learning weights. In a fully connected layer, each neuron is connected to all neurons in the previous layer, so it can learn complex relationships in the input data. Fully connected layers are often used in tasks such as image classification, object detection, and natural language processing.

[0152] The masking value is used to represent the energy distribution of each sound source at different frequencies in the mixed audio, that is, the energy distribution of the audio data of different speakers and the audio data of noise in the frame data.

[0153] The masking value may be an ideal ratio mask (IRM) or an ideal binary mask (IBM). The specific type of the masking value is not limited in the embodiments of the present application.

[0154] IRM can generate masking values ​​by calculating the ideal ratio of mixed audio and each sound source, so that the energy distribution of the separated sound sources is as close to the ideal ratio as possible. When applying IRM, an energy ratio estimate can be obtained by performing a short-time Fourier transform on the mixed audio and combining it with the spectral information of the audio source. This ratio is then used to construct a separation masking value, and finally the spectrum is masked to separate the sound sources.

[0155] IBM constrains the estimation of masking values ​​to binary form, that is, it separates the sound source into presence or absence at each time-frequency point. The masking value is generated by calculating the signal-to-noise ratio or energy ratio between the mixed audio and each sound source. When applying IBM, it can be determined whether to separate a sound source based on the estimated signal-to-noise ratio or ratio, and a binary masking operation can be performed on the spectrogram.

[0156] like Figure 6As shown, after the first electronic device obtains the masking value of the audio data of each speaker and the masking value of the audio data of the noise in the frame data, the first electronic device can obtain the audio data of each speaker and the audio data of the noise according to the frame data (i.e., the original MIC signal), the masking value of the audio data of each speaker and the masking value of the audio data of the noise. For example, the frame data (i.e., the original MIC signal) can be multiplied by the masking value of the audio data of each speaker and the masking value of the audio data of the noise, so as to obtain the audio data of each speaker and the audio data of the noise in the frame data, such as sound source 1, sound source 2, and sound source 3.

[0157] It should be noted that Figure 6 The process of separating the sound source of a frame of data is as follows: for each of the plurality of frame data into which the audio data sent by the second electronic device is divided, Figure 6 The process in is used to separate the sound source, thereby obtaining the audio data of each speaker and the audio data of the noise in each frame data.

[0158] Since each frame data may include audio data of different speakers and audio data of noise, the audio data of different speakers in different frame data may be the audio data of the same speaker. Therefore, after the first electronic device obtains the audio data of different speakers and audio data of noise in each frame data, the first electronic device can determine the audio data of the same speaker and audio data of noise in different frame data. Thus, based on the audio data of the same speaker in different frame data, the audio data of different speakers and audio data of noise in the audio data sent by the second electronic device can be determined.

[0159] In some examples, after the first electronic device obtains the audio data of different speakers and the audio data of noise in each frame data, the first electronic device can determine the audio data of the same speaker and the audio data of noise in different frame data based on the voiceprint features corresponding to the audio data of each speaker in each frame data and the voiceprint features corresponding to the audio data of noise.

[0160] The following is combined with Figure 7 An exemplary description is given of the process in which the first electronic device in an embodiment of the present application determines the audio data of the same speaker and the audio data of noise in different frame data based on the voiceprint features corresponding to the audio data of each speaker in each frame data (for each speaker in the embodiment of the present application, it can also be referred to as the first user, and the voiceprint features of the first user can also be referred to as the first voiceprint features) and the voiceprint features corresponding to the audio data of the noise.

[0161] like Figure 7As shown, for each frame data of the multiple frame data divided by the audio data sent by the second electronic device, after the first electronic device determines the audio data of each speaker and the audio data of the noise in the frame data, the first electronic device can perform feature extraction on the audio data of each speaker and the audio data of the noise in the frame data. That is, the first electronic device performs voiceprint feature extraction on the audio data of each speaker and the audio data of the noise in the frame data, thereby obtaining the voiceprint feature corresponding to the audio data of each speaker and the voiceprint feature corresponding to the audio data of the noise in the frame data.

[0162] Voiceprint features are characteristic data of different voices. Each person's voice is unique, including characteristics such as pitch, speaking speed, intonation, and pronunciation.

[0163] like Figure 7 As shown, after the first electronic device obtains the voiceprint features corresponding to the audio data of each speaker in the frame data and the voiceprint features corresponding to the audio data of the noise, the voiceprint features of the high-dimensional separable space corresponding to the audio data of each speaker in the frame data and the voiceprint features of the high-dimensional separable space corresponding to the audio data of the noise can be obtained through a convolutional neural network (CNN).

[0164] After the first electronic device obtains the voiceprint features of the high-dimensional separable space corresponding to the audio data of each speaker in the frame data, and the voiceprint features of the high-dimensional separable space corresponding to the audio data of the noise, it is separated through a long short term memory (LSTM) network to obtain the voiceprint features of the high-dimensional separable space corresponding to the audio data of each speaker in the frame data, and the voiceprint features of the high-dimensional separable space corresponding to the audio data of the noise.

[0165] Afterwards, the first electronic device obtains the voiceprint features of the high-dimensional separable space corresponding to the audio data of each speaker in the frame data, and the voiceprint features of the high-dimensional separable space corresponding to the audio data of the noise, through a fully connected layer (FC) to obtain the voiceprint features of the low-dimensional space corresponding to the audio data of each speaker in the frame data, and the voiceprint features of the low-dimensional space corresponding to the audio data of the noise.

[0166] In some examples, the voiceprint features of the low-dimensional space corresponding to the audio data of each speaker and the voiceprint features of the low-dimensional space corresponding to the audio data of the noise can be embedded signals. In audio processing, the embedding signal can convert the audio signal into a vector representation to facilitate the computer to perform tasks such as audio classification, speech recognition, and music generation. By mapping the audio signal to a low-dimensional space, the key features of the audio can be extracted and the similarities and differences between the audio can be captured.

[0167] After the first electronic device obtains the voiceprint features of the low-dimensional space corresponding to the audio data of each speaker in the frame data and the voiceprint features of the low-dimensional space corresponding to the audio data of the noise, the first electronic device can match the voiceprint features of the low-dimensional space corresponding to the audio data of each speaker in the frame data and the voiceprint features of the low-dimensional space corresponding to the audio data of the noise with the voiceprint features in the voiceprint library (which may also be referred to as the second voiceprint features in the embodiment of the present application), thereby determining whether the speaker in the frame data is a target person. The target person is the speaker corresponding to the voiceprint features in the voiceprint library (which may also be referred to as the second user in the embodiment of the present application).

[0168] In some examples, the voiceprint features in the voiceprint library may be embedded signals. The specific form of the voiceprint features is not limited in the embodiments of the present application, and the embodiments of the present application are illustrated by taking the voiceprint features as embedded signals as an example.

[0169] When the first electronic device determines that the speaker in the frame data is a target person, the first electronic device can determine the audio data of the target person in the frame data as the audio data of the same target person. For example, when the first electronic device determines that the voiceprint features of speaker A, speaker B, and speaker C exist in the frame data, and the voiceprint features in the voiceprint library include the voiceprint features of speaker A, speaker B, and speaker C, the first electronic device can determine that the speaker in the frame data is a target person (i.e., speaker A, speaker B, and speaker C).

[0170] When the first electronic device determines that the target person does not exist among the speakers in the frame data, the first electronic device can determine the speaker who is not the target person in the frame data as a new target person, and add the voiceprint features of the new target person to the voiceprint library. Therefore, when the first electronic device determines whether the speaker in the next frame data of the current frame is the target person, the new target person can be used for determination. For example, when the first electronic device determines that the voiceprint features of speaker A, speaker B, and speaker C exist in the frame data, and the voiceprint features in the voiceprint library include the voiceprint features of speaker A and speaker B, the first electronic device can determine that the target person (i.e., speaker A and speaker B) exists among the speakers in the frame data. At the same time, the first electronic device can determine that speaker C is determined as the new target person, and add the voiceprint features of speaker C to the voiceprint library.

[0171] It should be noted that the voiceprint library can be empty initially, and the voiceprint features in the voiceprint library can be obtained based on the voiceprint features in the low-dimensional space corresponding to the audio data of each speaker in the frame data, and the voiceprint features in the low-dimensional space corresponding to the audio data of the noise.

[0172] For example, after the audio data sent by the first electronic device to the second electronic device is divided into multiple frame data and the audio data of different speakers and the audio data of noise in each frame data are determined, the first electronic device can first determine whether there is a target person among the speakers in the first frame data in chronological order.

[0173] When the first electronic device determines that the voiceprint features of speaker A, speaker B, and speaker C exist in the first frame data, at this time, the voiceprint library is initially empty, that is, the first electronic device determines that the target person does not exist among the speakers in the first frame data. Then the first electronic device stores the voiceprint features of speaker A, speaker B, and speaker C in the voiceprint library. Afterwards, the first electronic device determines whether the speaker in the second frame data exists as the target person. When the first electronic device determines that the voiceprint features of speaker A and speaker C exist in the second frame data, at this time, the voiceprint library includes the voiceprint features of speaker A, speaker B, and speaker C, that is, the first electronic device determines that the target person (i.e., speaker A and speaker C) exists among the speakers in the second frame data.

[0174] At the same time, the first electronic device may determine the audio data of speaker A in the first frame data and the audio data of speaker A in the second frame data as the audio data of speaker A in the audio data sent by the second electronic device, and store the audio data of speaker A and the time information corresponding to the audio data of speaker A (i.e., the time information of the audio data of speaker A in the first frame data and the time information of the audio data of speaker A in the second frame data). The first electronic device may determine the audio data of speaker B in the first frame data (which may also be referred to as sub-audio data in the embodiment of the present application) as the audio data of speaker B in the audio data sent by the second electronic device, and store the audio data of speaker B and the time information corresponding to the audio data of speaker B (i.e., the time information of the audio data of speaker B in the first frame data). The first electronic device can determine the audio data of speaker C in the first frame data and the audio data of speaker C in the second frame data as the audio data of speaker C in the audio data sent by the second electronic device, and store the audio data of speaker C and the event information corresponding to the audio data of speaker C (i.e., the time information of the audio data of speaker C in the first frame data and the time information of the audio data of speaker C in the second frame data).

[0175] Afterwards, the first electronic device determines whether the speaker in the third frame data is the target person. When the first electronic device determines that there is a voiceprint feature of noise in the third frame data, the voiceprint library includes the voiceprint features of speaker A, the voiceprint features of speaker B, and the voiceprint features of speaker C, that is, the first electronic device determines that there is no target person among the speakers in the third frame data. Then the first electronic device stores the voiceprint features of noise in the third frame data in the voiceprint library. The audio data of noise in the third frame data is determined as the audio data of noise in the audio data sent by the second electronic device, and the audio data of noise and the time information corresponding to the audio data of noise (that is, the time information of the audio data of noise in the third frame data) are stored.

[0176] Afterwards, the first electronic device can perform the above-mentioned determination process on each frame data, and update the voiceprint library according to whether the target person exists in each frame data. After the first electronic device completes the above-mentioned determination process on each frame data, the first electronic device can determine the audio data and corresponding time information of each speaker, and the audio data and corresponding time information of noise included in the audio data sent by the second electronic device. The audio data of the speaker, that is, the audio data of the speaker in different frame data. The time information corresponding to the audio data of the speaker, that is, the time information corresponding to the audio data of the speaker in different frame data. The audio data of noise, that is, the audio data of noise in different frame data. The time information corresponding to the audio data of noise, that is, the time information corresponding to the audio data of noise in different frame data.

[0177] In some examples, the first electronic device determines the audio data of each speaker based on the audio data sent by the second electronic device, and the HAL layer of the first electronic device (such as the audio module included in the HAL layer) can determine the audio data of each speaker and the audio data of noise in the audio data sent by the second electronic device based on the audio data sent by the second electronic device. Thus, the HAL layer (such as the audio module included in the HAL layer) can determine the audio data of the main speaker based on the audio data of each speaker.

[0178] S503: The first electronic device determines the audio data of the main speaker based on the audio data of each speaker.

[0179] After the first electronic device determines the audio data of each speaker in the audio data sent by the second electronic device based on the audio data sent by the second electronic device, the first electronic device can determine the audio data of the main speaker (i.e., the audio data of the target user) based on the audio data of each speaker. The main speaker can be a speaker who meets the preset conditions in the audio data sent by the second electronic device. The preset conditions can include at least one of the highest speaking probability, the longest speaking time, or the highest speaking energy.

[0180] In some examples, the first electronic device determines the audio data of the main speaker based on the audio data of each speaker, which may include: the first electronic device determines the main speaker based on the audio data of each speaker. The main speaker may be the speaker with the highest speaking probability, the longest speaking time, or the greatest speaking energy in the audio data sent by the second electronic device.

[0181] In some examples, when a first electronic device establishes a video call with a second electronic device and the first electronic device can simultaneously receive audio data and image data sent by the second electronic device, the first electronic device determines the audio data of the main speaker based on the audio data of each speaker. The first electronic device can determine the main speaker based on the audio data of each speaker in combination with the image data.

[0182] In some examples, the first electronic device can determine the time information of each person speaking based on the image data and the time information corresponding to the image data. For example, the first electronic device can determine the lip movement information of each speaker based on the image data and the time information corresponding to the image data. The lip movement information of each speaker may include the time information when each speaker moves his lips. Afterwards, the first electronic device can determine the speaking time information of each speaker based on the lip movement information of each speaker. Afterwards, the first electronic device can determine the speaker with the longest speaking time, that is, the main speaker, based on the speaking time information of each speaker and the time information corresponding to the audio data sent by the second electronic device.

[0183] In other examples, the first electronic device may determine the audio data of the main speaker based on the image data and time information corresponding to the image data, and the audio data of each speaker and the corresponding time information.

[0184] For example, combined with Figure 8 As shown, taking the example of a first electronic device receiving audio data and image data sent by a second electronic device, an example is given of a process in which the first electronic device determines the audio data of the main speaker based on the image data and the time information corresponding to the image data, the audio data of each speaker and the corresponding time information.

[0185] like Figure 8As shown, after the first electronic device receives the audio data and image data sent by the second electronic device, the first electronic device determines, based on the audio data sent by the second electronic device, that the audio data sent by the second electronic device includes the audio data of speaker A and the audio data of speaker B, that is, the audio data of speaker A includes the audio data of speaker A in different frame data and the corresponding time information, and the audio data of speaker B includes the audio data of speaker A in different frame data and the corresponding time information.

[0186] After the first electronic device obtains the audio data of speaker A and the audio data of speaker B, the first electronic device may pass the audio data of speaker A through a convolutional neural network (CNN) to obtain an embedding signal corresponding to the audio data of speaker A and an embedding signal corresponding to the time information corresponding to the audio data of speaker A. At the same time, the first electronic device passes the audio data of speaker B through a convolutional neural network (CNN) to obtain an embedding signal corresponding to the audio data of speaker B and an embedding signal corresponding to the time information corresponding to the audio data of speaker B.

[0187] like Figure 8 As shown, while the first electronic device obtains the embedding signal corresponding to the audio data of speaker A and the embedding signal corresponding to the time information corresponding to the audio data of speaker A, and obtains the embedding signal corresponding to the audio data of speaker B and the embedding signal corresponding to the time information corresponding to the audio data of speaker B, the first electronic device can obtain the embedding signal corresponding to the image data sent by the second electronic device.

[0188] like Figure 8 As shown, when the first electronic device obtains the embedding signal corresponding to the image data sent by the second electronic device, the image data can be first framed according to the preset duration. The preset duration may be the same as or different from the preset duration for the first electronic device to divide the audio data sent by the second electronic device into multiple frames of data. This is not limited in the embodiments of the present application. In the embodiments of the present application, an example is given in which the preset duration for the first electronic device to frame the image data sent by the second electronic device is the same as the preset duration for the first electronic device to divide the audio data sent by the second electronic device into multiple frames of data. For example, when the preset duration for the first electronic device to divide the audio data sent by the second electronic device into multiple frames of data is 5 milliseconds, the preset duration for the first electronic device to frame the image data sent by the second electronic device is 5 milliseconds.

[0189] After the first electronic device divides the image data sent by the second electronic device into a plurality of image frame data, the first electronic device can process each image frame data through a convolutional neural network (CNN) and a deep convolutional neural network (RESNET50) according to the time information of each image frame data (i.e., the sequence of each image frame data), thereby obtaining an embedding signal corresponding to each image frame data. The embedding signal corresponding to each image frame data can be used to determine the lip movement information of different speakers.

[0190] Afterwards, the first electronic device can pass the embedding signal corresponding to the audio data of speaker A and the embedding signal corresponding to the time information corresponding to the audio data of speaker A, the embedding signal corresponding to the audio data of speaker B and the embedding signal corresponding to the time information corresponding to the audio data of speaker B, and the embedding signal corresponding to the image data sent by the second electronic device through a separation network (conformer), thereby obtaining the probability of different speakers in the audio data sent by the second electronic device. For example, the speaking duration of speaker A is obtained based on the time information of the lip movement of speaker A in different image frame data. The probability of speaker A can be determined by comparing the speaking duration of speaker A with the duration of the audio data sent by the first electronic device.

[0191] After the first electronic device obtains the probabilities of different speakers, the first electronic device can determine the speaker with the highest probability as the main speaker (i.e., the main speaker). After the main speaker is determined, the audio data of the main speaker is the audio data of the main speaker. Speakers with a lower probability than the main speaker are other speakers.

[0192] It should be noted that before the first electronic device passes the embedding signal corresponding to the audio data of speaker A and the embedding signal corresponding to the time information corresponding to the audio data of speaker A, the embedding signal corresponding to the audio data of speaker B and the embedding signal corresponding to the time information corresponding to the audio data of speaker B, and the embedding signal corresponding to the image data sent by the second electronic device through a separation network (conformer) to obtain the probability of different speakers in the audio data sent by the second electronic device, the first electronic device can use the sample data to train the separation network (conformer). The sample data may include the embedding signals corresponding to the audio data of different speakers, the embedding signals corresponding to the time information corresponding to the audio data of different speakers, the embedding signals corresponding to the image data, and the probabilities of different speakers.

[0193] In other examples, when the first electronic device receives audio data sent by the second electronic device but no image data, the first electronic device can determine the score of each speaker based on the audio data of each speaker. The score of the speaker can be used to indicate the duration of the speaker's speech and the degree of energy (i.e., sound energy). Afterwards, the first electronic device can determine the audio data of the speaker with the largest score as the main speaker.

[0194] For example, combined with Fig. 9 As shown, taking the case where a first electronic device receives audio data sent by a second electronic device as an example, an example is given of a process in which the first electronic device determines the score of each speaker based on the audio data of each speaker, and determines the audio data of the main speaker based on the score of each speaker.

[0195] like Fig. 9 As shown, after the first electronic device receives the audio data sent by the second electronic device, the first electronic device determines, based on the audio data sent by the second electronic device, that the audio data sent by the second electronic device includes the audio data of speaker A, the audio data of speaker B, and the audio data of speaker C, that is, the audio data of speaker A includes the audio data of speaker A in different frame data and the corresponding time information, the audio data of speaker B includes the audio data of speaker A in different frame data and the corresponding time information, and the audio data of speaker C includes the audio data of speaker C in different frame data and the corresponding time information.

[0196] After the first electronic device obtains the audio data of speaker A, the audio data of speaker B, and the audio data of speaker C, the first electronic device determines the energy and duration of speaker A, the energy and duration of speaker B, and the energy and duration of speaker C, respectively, based on the audio data of speaker A, the audio data of speaker B, and the audio data of speaker C. Afterwards, the first electronic device can determine the score of speaker A based on the energy and duration of speaker A and the weight of speaker A, determine the score of speaker B based on the energy and duration of speaker B and the weight of speaker B, and determine the score of speaker C based on the energy and duration of speaker C and the weight of speaker C. The weights of different speakers can be the same or different, and this is not limited in the embodiments of the present application. In the embodiments of the present application, the same weights of different speakers are used as an example for schematic description.

[0197] For example, the first electronic device may determine the scores of different speakers according to the following formula 1.

[0198] Formula 1: A = a*energy + (1-a)*duration;

[0199] Among them, A can represent the score of the same speaker, a can represent the weight of the speaker, energy is the energy of the speaker in the audio data sent by the second electronic device, and duration is the duration of the speaker in the audio data sent by the second electronic device.

[0200] It should be noted that, when a first electronic device establishes a video call with a second electronic device and the first electronic device simultaneously receives audio data and image data sent by the second electronic device, the first electronic device can determine the main speaker based on the audio data of each speaker in combination with the image data. The first electronic device can also determine the main speaker based on the audio data of each speaker in combination with the image data, and then determine the main speaker based on the score of each speaker.

[0201] When the main speaker determined by the first electronic device based on the audio data of each speaker in combination with the image data is different from the main speaker determined by the first electronic device based on the score of each speaker, the main speaker determined by the first electronic device based on the audio data of each speaker in combination with the image data may prevail.

[0202] In some examples, the first electronic device determines the audio data of the main speaker based on the audio data of each speaker, and the HAL layer of the first electronic device (such as the audio module included in the HAL layer) determines the audio data of the main speaker based on the audio data of each speaker. Thus, the HAL layer (such as the audio module included in the HAL layer) can enhance the audio data of the main speaker and suppress the audio data of other speakers.

[0203] S504: The first electronic device performs enhancement processing on the audio data of the main speaker, and performs suppression processing on the audio data of other speakers.

[0204] After the first electronic device determines the audio data of the main speaker, the first electronic device can perform sound effect processing on the audio data of the main speaker (i.e., the audio data of the target user) and the audio data of other speakers (i.e., the audio data of non-target users) respectively, so that when the first electronic device outputs the audio data of the main speaker and the audio data of the main speaker at the same time, it can ensure that the user can pay attention to the voices of other speakers while hearing the audio data of the main speaker clearly.

[0205] It should be noted that after the first electronic device determines the audio data of the main speaker, if the audio data sent by the second electronic device includes audio data of noise, the first electronic device may also perform sound effect processing on the audio data of noise. The sound effect processing method for the audio data of noise may be the same as the sound effect processing method for the audio data of other speakers by the first electronic device, such as spatial position separation and different gain control. The sound effect processing method for the audio data of noise may also be different from the sound effect processing method for the audio data of other speakers by the first electronic device, such as the sound effect processing method for the audio data of other speakers may be spatial position separation and different gain control, and the sound effect processing method for the audio data of noise may be elimination processing.

[0206] In some examples, the first electronic device performs sound effect processing (i.e., sound effect enhancement processing) on ​​the audio data of the main speaker and the audio data of other speakers respectively, which may include the first electronic device spatially separating the audio data of the main speaker and the audio data of other speakers, thereby enhancing the audio data of the main speaker and suppressing the audio data of other speakers.

[0207] For example, the first electronic device may set the transmission function corresponding to the audio data of the main speaker to the transmission function corresponding to the direction angle of 0 degrees (which may also be referred to as the first direction angle in the embodiment of the present application), and set the transmission function corresponding to the audio data of other speakers to the transmission function corresponding to the direction angle of other angles (such as 30 degrees, 60 degrees, etc., which may also be referred to as the second direction angle in the embodiment of the present application), so that when the first electronic device transmits the audio data of the main speaker according to the transmission function corresponding to the direction angle of 0 degrees, the user can feel that the voice of the main speaker is heard directly in front of the human ear. When the first electronic device transmits the audio data of other speakers according to the transmission function corresponding to the direction angle of other angles (such as 30 degrees, 60 degrees, etc.), the user can feel that the voice of other speakers is heard at other positions corresponding to the human ear. That is, the audio data of the main speaker and the audio data of other speakers are separated in spatial positions, so that the user feels that the voice of the main speaker is heard directly in front of the human ear, thereby achieving the enhancement processing of the audio data of the main speaker and the suppression processing of the audio data of other speakers.

[0208] It should be noted that, when there are multiple other speakers, the transmission functions corresponding to different other speakers may be set as transmission functions corresponding to the same direction angle, or may be set as transmission functions corresponding to different direction angles.

[0209] When the transmission functions corresponding to different other speakers are set as transmission functions corresponding to different angles, the secondary speaker and the ordinary speaker can be determined according to the highest speaking probability, longest speaking time or speaking energy of the other speakers. For example, the speaking probability of the secondary speaker can be less than the speaking probability of the main speaker, and greater than the speaking probability of the ordinary speaker. Alternatively, the speaking energy of the secondary speaker can be less than the speaking energy of the main speaker, and greater than the speaking energy of the ordinary speaker. Alternatively, the speaking time of the secondary speaker can be less than the speaking time of the main speaker, and greater than the speaking time of the ordinary speaker. Afterwards, the transmission function corresponding to the secondary speaker can be set as a transmission function whose direction angle is greater than the direction angle corresponding to the main speaker, and less than the direction angle corresponding to the ordinary speaker.

[0210] For example, when the main speaker is speaker A, and other speakers include speaker B and speaker C, the first electronic device determines that the speaking time of speaker B is less than the speaking time of speaker A, and greater than the speaking time of speaker C, the first electronic device can determine that speaker B is a secondary speaker and speaker C is a common speaker. Afterwards, the first electronic device can transmit the audio data of speaker B according to the transmission function corresponding to the direction angle of 0 degrees, transmit the audio data of speaker B according to the transmission function corresponding to the direction angle of 30 degrees, and transmit the audio data of speaker C according to the transmission function corresponding to the direction angle of 60 degrees.

[0211] When the audio data sent by the second electronic device includes the audio data of noise, the transmission function corresponding to the audio data of noise can be set to the transmission function corresponding to the same angle as other speakers, or can be set to the transmission function corresponding to the angle different from other speakers. For example, when the audio data sent by the second electronic device includes the audio data of noise, and other speakers include speakers B and speakers C, the first electronic device transmits the audio data of speaker B according to the transmission function corresponding to the direction angle of 30 degrees, transmits the audio data of speaker C according to the transmission function corresponding to the direction angle of 30 degrees, and transmits the audio data of noise according to the transmission function corresponding to the direction angle of 30 degrees. The first electronic device can also transmit the audio data of speaker B according to the transmission function corresponding to the direction angle of 30 degrees (which can also be called the first sub-direction angle in the embodiment of the present application), transmit the audio data of speaker C according to the transmission function corresponding to the direction angle of 60 degrees (which can also be called the second sub-direction angle in the embodiment of the present application), and transmit the audio data of noise according to the transmission function corresponding to the direction angle of 90 degrees.

[0212] The transfer function corresponding to the audio data is the head-related transfer function. The head-related impulse response (HRIR) is a function that describes the filtering and phase difference effects of sound signals when they pass through the head and ears. HRIR represents the time and amplitude characteristics of the sound source reaching the listener's cochlea through the head and ears at different directions and distances. By convolving the sound source signal with the appropriate HRIR, a stereo and directional sound effect can be simulated. HRIR data is usually obtained by placing a miniature microphone array on a simulated or actual human head for measurement. Each measurement point (direction and distance) records the effect of the head on the sound and generates corresponding HRIR data.

[0213] The head-related transfer function (HRIR) may include a transfer function of the left ear and a transfer function of the right ear. The head-related transfer function (HRIR) may be expressed by the following formulas 2 and 3.

[0214] Formula 2: outL=s*hrir_l(azim,elev);

[0215] Among them, outL represents the sound source signal of the left ear, s represents the audio data of different speakers, azim represents the direction angle, elev represents the altitude angle, and hrir_l(azim,elev) represents the head-related transfer function corresponding to different direction angles and altitude angles.

[0216] Formula 3: outR = s*hrir_r (azim, elev);

[0217] Among them, outR represents the sound source signal of the left ear, s represents the audio data of different speakers, azim represents the direction angle, elev represents the altitude angle, and hrir_r(azim,elev) represents the head-related transfer function corresponding to different direction angles and altitude angles.

[0218] In the example of the present application, the elevation angle can be set to the same elevation angle or different elevation angles for different speakers. This is not limited in the embodiment of the present application. In the embodiment of the present application, the elevation angle is set to the same elevation angle, such as an elevation angle of 0 degrees, for example.

[0219] That is to say, the first electronic device may pre-store transmission functions corresponding to different direction angles (the transmission functions corresponding to different direction angles include the transmission function of the left ear and the transmission function of the right ear corresponding to the same direction angle). When the first electronic device determines the audio data of the main speaker and the audio data of other speakers (or the audio data of noise), the first electronic device may select the transmission function corresponding to the main speaker and the transmission functions corresponding to the audio data of other speakers (or the audio data of noise) from the transmission functions pre-stored corresponding to different direction angles.

[0220] In other examples, the first electronic device performs sound effect processing (i.e., sound effect suppression processing) on ​​the audio data of the main speaker and the audio data of other speakers respectively, and the gain attenuation of the audio data of the main speaker can be set to be smaller than the gain attenuation of the audio data of other speakers for the first electronic device. That is, different gain attenuations are set for the audio data of the main speaker and the audio data of other speakers, so that the user can hear the voice of the main speaker clearly and the voices of other speakers at the same time, thereby achieving enhancement processing of the audio data of the main speaker and suppression processing of the audio data of other speakers.

[0221] Gain attenuation mainly refers to the amplitude change of audio data during transmission, that is, the reduction in the loudness of audio data. During the transmission of audio data, the audio data will gradually attenuate due to the loss and interference of audio data. In order to overcome this attenuation, the audio data can be appropriately amplified or reduced, that is, the gain can be adjusted to maintain the stability and audibility of the audio signal. In other words, the greater the gain attenuation, the smaller the amplitude change of the audio signal during transmission, that is, the smaller the reduction in the loudness of the audio signal. The smaller the gain attenuation, the greater the amplitude change of the audio signal during transmission, that is, the greater the reduction in the loudness of the audio signal.

[0222] In some examples, the first electronic device may set the gain attenuation corresponding to the gain attenuation of the audio data of the main speaker to a first coefficient, and set the coefficient corresponding to the gain attenuation of the audio data of other speakers to a second coefficient, where the second coefficient is greater than the first coefficient, so that the gain attenuation of the audio data of the main speaker determined by the first coefficient is greater than the gain attenuation of the audio data of other speakers determined by the second coefficient.

[0223] When the other speakers are multiple speakers, the gain attenuations corresponding to different other speakers may be set to the same gain attenuation or different gain attenuations.

[0224] When the gain attenuation corresponding to different other speakers is set to different gain attenuations, the secondary speaker and the ordinary speaker can be determined according to the highest speaking probability, longest speaking time or speaking energy of the other speakers. For example, the speaking probability of the secondary speaker can be less than the speaking probability of the main speaker, and greater than the speaking probability of the ordinary speaker. Alternatively, the speaking energy of the secondary speaker can be less than the speaking energy of the main speaker, and greater than the speaking energy of the ordinary speaker. Alternatively, the speaking time of the secondary speaker can be less than the speaking time of the main speaker, and greater than the speaking time of the ordinary speaker. Afterwards, the gain attenuation corresponding to the secondary speaker can be set to be less than the gain attenuation corresponding to the main speaker, and greater than the gain attenuation corresponding to the ordinary speaker.

[0225] For example, when the main speaker is speaker A, and other speakers include speaker B and speaker C, the first electronic device determines that the speaking time of speaker B is less than the speaking time of speaker A and greater than the speaking time of speaker C, and the first electronic device can determine that speaker B is the secondary speaker and speaker C is the ordinary speaker. Afterwards, the first electronic device can set the gain attenuation coefficient corresponding to the audio data of the main speaker (i.e., speaker A) to s=0.5, the gain attenuation coefficient corresponding to the audio data of the secondary speaker (i.e., speaker B) to s=1, and the gain attenuation coefficient corresponding to the audio data of the ordinary speaker (i.e., speaker C) to s=1.5.

[0226] Afterwards, the first electronic device may determine the gain attenuation of different speakers according to the following formula 4.

[0227] Formula 4: factor = 1 / (1+s^2);

[0228] Among them, factor can represent the gain attenuation of different speakers, s can represent the gain attenuation coefficient corresponding to different speakers, and s^2 can represent the square of the gain attenuation coefficient corresponding to different speakers.

[0229] The first gain attenuation corresponding to the gain attenuation coefficient s=0.5 is greater than the second gain attenuation corresponding to the gain attenuation coefficient s=1.

[0230] It should be noted that when the audio data sent by the second electronic device includes audio data of noise, the gain attenuation corresponding to the audio data of noise can be set to the same gain attenuation as other speakers, or to a gain attenuation different from other speakers. For example, when the audio data sent by the second electronic device includes audio data of noise, and other speakers include speaker B and speaker C, the first electronic device can set the gain attenuation of speaker B's audio data to the gain attenuation corresponding to the gain attenuation coefficient of 1, set the gain attenuation of speaker C's audio data to the gain attenuation corresponding to the gain attenuation coefficient of 1.5, and set the gain attenuation of the audio data of noise to the gain attenuation corresponding to the gain attenuation coefficient of 1.5. The first electronic device can also set the gain attenuation of speaker B's audio data to the gain attenuation corresponding to the gain attenuation coefficient of 1 (which may also be referred to as the first sub-gain attenuation in the embodiment of the present application), set the gain attenuation of speaker C's audio data to the gain attenuation corresponding to the gain attenuation coefficient of 1.5 (which may also be referred to as the second sub-gain attenuation in the embodiment of the present application), and set the gain attenuation of the audio data of noise to the gain attenuation corresponding to the gain attenuation coefficient of 2.

[0231] In some examples, the first electronic device performs sound effect processing on the audio data of the main speaker and the audio data of other speakers respectively, which may include: the first electronic device spatially separates the audio data of the main speaker and the audio data of other speakers, and sets the gain attenuation of the audio data of the main speaker to be smaller than the gain attenuation of the audio data of other speakers.

[0232] In some examples, the first electronic device enhances the audio data of the main speaker and suppresses the audio data of other speakers, which can be the HAL layer of the first electronic device (such as the audio module included in the HAL layer). The first electronic device enhances the audio data of the main speaker and suppresses the audio data of other speakers. Thus, the HAL layer (such as the audio module included in the HAL layer) can obtain the enhanced audio data of the main speaker and the suppressed audio data of other speakers.

[0233] S505: The first electronic device simultaneously outputs the enhanced audio data of the main speaker and the suppressed audio data of other speakers.

[0234] After the first electronic device enhances the audio data of the main speaker in the audio data sent by the second electronic device, and suppresses the audio data of other speakers in the audio data sent by the second electronic device, the first electronic device can simultaneously output the enhanced audio data of the main speaker and the suppressed audio data of other speakers, that is, output the target audio data.

[0235] In some examples, after the first electronic device enhances the audio data of the main speaker in the audio data sent by the second electronic device, and suppresses the audio data of other speakers in the audio data sent by the second electronic device, the first electronic device can accumulate the enhanced audio data of the main speaker and the suppressed audio data of other speakers according to the time information to obtain the final spatial virtual signal. Thus, the first electronic device can output the accumulated spatial virtual signal. For example, based on the time information of the audio data of the main speaker and the time information of the audio data of other speakers, the speakers at different times can be determined, and the audio data of different speakers after sound effect processing can be accumulated and output at different times.

[0236] The solution of the present application is that the first electronic device can determine the audio data of the main speaker in the audio data, as well as the audio data of other speakers and the audio data of noise based on the audio data sent by the second electronic device, that is, the first electronic device can dynamically determine the audio data of the main speaker in the audio data sent by the second electronic device, as well as the audio data of other speakers and the audio data of noise. Therefore, the first electronic device can determine the audio data of the main speaker in real time, so that after the main speaker changes, the first electronic device still enhances and outputs the audio data of the changed main speaker, ensuring that the user can hear the audio data of the changed main speaker clearly. Moreover, the solution of the present application is that the first electronic device can simultaneously output the audio data of the main speaker after the enhancement processing and the audio data of other speakers and the audio data of noise after the suppression processing, which can ensure that the user can pay attention to the voices of other speakers while hearing the audio data of the main speaker clearly.

[0237] For ease of understanding, the following Fig.10 The audio processing method provided in the embodiment of the present application is described. Fig.10 As shown, the audio processing method may include the following S1001-S1005.

[0238] S1001. A first electronic device obtains call data collected by a second electronic device, where the call data includes first audio data, and the first audio data includes audio data of multiple users.

[0239] After the first electronic device establishes a call with the second electronic device, the first electronic device can obtain the call data collected by the second electronic device. The call data may include first audio data, and the first audio data may include audio data of multiple users (ie, multiple speakers and noise).

[0240] The call data may also include first image data corresponding to the first audio data. That is, when the first electronic device establishes a voice call with the second electronic device, the first electronic device can obtain the first audio data collected by the second electronic device. When the first electronic device establishes a video call with the second electronic device, the first electronic device can obtain the first audio data collected by the second electronic device and the first image data corresponding to the first audio data.

[0241] Specifically, the specific implementation method of the first electronic device obtaining the call data collected by the second electronic device in the embodiment of the present application can refer to the specific implementation method in the above S501, which will not be repeated in the embodiment of the present application.

[0242] S1002: The first electronic device determines audio data of each user among a plurality of users based on the first audio data.

[0243] After the first electronic device obtains the call data collected by the second electronic device, the first electronic device can determine the audio data of each of the multiple users according to the first audio data, that is, determine the audio data of different speakers and the audio data of noise.

[0244] In some examples, the first electronic device determines the audio data of each user among the multiple users based on the first audio data, which may include: the first electronic device performs frame processing on the first audio data to obtain multiple audio frame data. Then, the first electronic device performs sound source separation on the audio frame data to obtain sub-audio data of each user in the audio frame data. Then, the first electronic device obtains the audio data of each user based on the sub-audio data of each user in the multiple audio frame data.

[0245] Since the first electronic device performs frame processing on the first audio data to obtain multiple audio frame data, the first electronic device can separate the sound sources of the audio frame data to obtain sub-audio data of each user in the audio frame data, and can accurately determine the audio data of each user in the first audio data collected by the second electronic device.

[0246] In some examples, the first electronic device obtains the audio data of each user based on the sub-audio data of each user in the multiple audio frame data, which may include: the first electronic device extracts the voiceprint feature of the sub-audio data of the first user to obtain the first voiceprint feature. The first user is any one of the multiple users. The first electronic device matches the first voiceprint feature with the second voiceprint feature in the voiceprint library. The second voiceprint feature is the voiceprint feature of the second user, and the second user is any one of the multiple users. In the case where the first voiceprint feature does not match the second voiceprint feature, the first electronic device determines that the first user and the second user are different users, and determines the sub-audio data of the first user as the audio data of the first user; the first electronic device stores the first voiceprint feature in the voiceprint library. In the case where the first voiceprint feature matches the second voiceprint feature, the first electronic device determines that the first user and the second user are the same user, and determines that the audio data of the second user includes the sub-audio number of the first user.

[0247] Since the first electronic device can extract the voiceprint features of the user and match the voiceprint features of the user with the voiceprint features in the voiceprint library, when no match is successful, the first electronic device can determine that the user is not in the voiceprint library, and at the same time, the first electronic device can update the voiceprint library according to the voiceprint features of the user. That is, the first electronic device can determine whether the user is a user in the voiceprint library according to the voiceprint features of the user and the voiceprint features in the voiceprint library.

[0248] When the match is successful, the first electronic device can determine that the user is an existing user in the voiceprint library, and thus can determine that the user's audio data is the audio data corresponding to the existing user. That is, the first electronic device can determine whether the user's audio data is the audio data of the same user based on the user's voiceprint features and the voiceprint features in the voiceprint library.

[0249] Specifically, in the embodiment of the present application, the specific implementation method of the first electronic device determining the audio data of each user among multiple users based on the first audio data can refer to the specific implementation method in the above S502, which will not be repeated in the embodiment of the present application.

[0250] S1003. The first electronic device determines second audio data of a target user and third audio data of non-target users according to audio data of each user; the non-target users include users other than the target user among the multiple users.

[0251] After the first electronic device determines the audio data of each user, the first electronic device can determine the second audio data of the target user and the third audio data of the non-target user according to the audio data of each user. The target user is the main speaker. The non-target user includes users other than the target user among the multiple users. That is, the non-target user may include a secondary speaker, and the non-target user may also include noise, such as the sound emitted by a device such as a television.

[0252] In some examples, the first electronic device determines the second audio data of the target user and the third audio data of the non-target user based on the audio data of each user, which may include: the first electronic device determines the speaking time and sound energy of each user based on the audio data of each user. The first electronic device determines the target user based on the speaking time and sound energy of each user. The target user is the user with the longest speaking time and / or the largest sound energy among multiple users. The first electronic device determines the audio data of the target user as the second audio data, and determines the audio data of the non-target user as the third audio data.

[0253] The first electronic device can accurately identify that the first audio data includes the audio data of the main speaker among the audio data of multiple users through the speaking duration and sound energy of each user. Thus, the first electronic device can perform sound enhancement processing on the audio data of the target user, so that when the first electronic device outputs the audio data of the target user, the user of the first electronic device can hear the voice of the target user clearly.

[0254] In other examples, when the call data also includes the first image data corresponding to the first audio data, the first electronic device determines the second audio data of the target user and the third audio data of the non-target user based on the audio data of each user, which may include: the first electronic device determines the lip movement information of each user in the first audio data based on the first image data. The first electronic device determines the probability of each user speaking based on the lip movement information of each user and the audio data of each user. The first electronic device determines the target user among multiple users based on the probability of each user speaking. The target user may be the user with the highest probability of speaking among multiple users. The first electronic device determines the audio data of the target user as the second audio data, and determines the audio data of the non-target user as the third audio data.

[0255] The first electronic device determines the lip movement information of each user based on the first image data corresponding to the first audio data, and can accurately determine the speaking probability of each user in combination with the lip movement information of each user. The user with the highest speaking probability among the speaking probabilities of each user is determined as the target user, and the target user can be accurately determined. Thus, the first electronic device can perform sound enhancement processing on the audio data of the target user, so that when the first electronic device outputs the audio data of the target user, the user of the first electronic device can hear the voice of the target user clearly.

[0256] Specifically, in the embodiment of the present application, the specific implementation method of the first electronic device determining the second audio data of the target user and the third audio data of the non-target user based on the audio data of each user can refer to the specific implementation method in the above S503, which will not be repeated in the embodiment of the present application.

[0257] S1004: The first electronic device performs sound effect enhancement processing on the second audio data to obtain fourth audio data, and performs sound effect suppression processing on the third audio data to obtain fifth audio data.

[0258] After the first electronic device determines the second audio data of the target user and the third audio data of the non-target user, the first electronic device performs sound enhancement processing on the second audio data to obtain fourth audio data, and performs sound suppression processing on the third audio data to obtain fifth audio data.

[0259] In some examples, the first electronic device performs sound enhancement processing on the second audio data to obtain fourth audio data, and performs sound suppression processing on the third audio data to obtain fifth audio data, which may include: the first electronic device performs sound enhancement processing on the second audio data according to the transfer function (also called head-related transfer function) corresponding to the first direction angle and / or the first gain attenuation to obtain the fourth audio data; the first electronic device performs sound suppression processing on the third audio data according to the transfer function corresponding to the second direction angle and / or the second gain attenuation to obtain the fifth audio data; wherein the first direction angle is smaller than the second direction angle, and / or the first gain attenuation is greater than the second gain attenuation.

[0260] In some examples, the first direction angle may be 0 degrees and the second direction angle may be 30 degrees.

[0261] Since the first electronic device can perform sound enhancement processing on the second audio data according to the transmission function corresponding to the first direction angle, and perform sound suppression processing on the third audio data according to the transmission function corresponding to the second direction angle, and the first direction angle is smaller than the second direction angle, the audio data of the target user and the audio data of the non-target user can be separated from the spatial position. Thus, the user of the first electronic device can hear the voice of the target user clearly and pay attention to the voice of the non-target user. Since the first electronic device can perform sound enhancement processing on the second audio data according to the first gain attenuation, and perform sound suppression processing on the third audio data according to the second gain attenuation, and the first gain attenuation is greater than the second gain attenuation coefficient, it can be ensured that the amplitude change of the audio data of the target user during the transmission process is smaller than the amplitude change of the audio data of the non-target user during the transmission process, thereby enabling the user of the first electronic device to hear the voice of the target user clearly and pay attention to the voice of the non-target user.

[0262] In some examples, when the non-target users include multiple users, such as a third user and a fourth user, the third audio data may include audio data of the third user and audio data of the fourth user, and the second direction angle may include a first sub-direction angle and a second sub-direction angle, the first sub-direction angle may be smaller than the second sub-direction angle, and the second sub-direction angle may be greater than the first direction angle.

[0263] In some examples, the first direction angle may be 0 degrees, the first sub-direction angle may be 30 degrees, and the second sub-direction angle may be 60 degrees.

[0264] The first electronic device performs sound effect suppression processing on the third audio data according to the transmission function corresponding to the second direction angle to obtain the fifth audio data, which may include: the first electronic device performs sound effect suppression processing on the audio data of the third user according to the transmission function corresponding to the first sub-direction angle to obtain the sixth audio data. The first electronic device performs sound effect suppression processing on the audio data of the fourth user according to the transmission function corresponding to the second sub-direction angle to obtain the seventh audio data; wherein the fifth audio data includes the sixth audio data and the seventh audio data.

[0265] Since the first electronic device can suppress the audio data of different users among the non-target users according to the transmission function corresponding to the first sub-direction angle and the transmission function corresponding to the second sub-direction angle, the audio data of different users among the non-target users can be further separated in spatial positions. Thus, the user of the first electronic device can hear the voice of the target user clearly and pay attention to the voices of different non-target users.

[0266] In other examples, for multiple non-target users, such as a fifth user and a sixth user, the third audio data may include audio data of the fifth user and audio data of the sixth user, and the second gain attenuation may include a first sub-gain attenuation coefficient and a second sub-gain attenuation, the first sub-gain attenuation being greater than the second sub-gain attenuation, and the first sub-gain attenuation being less than the first gain attenuation.

[0267] The first electronic device performs sound effect suppression processing on the third audio data according to the second gain attenuation to obtain the fifth audio data, which may include: the first electronic device performs sound effect suppression processing on the audio data of the fifth user according to the first sub-gain attenuation to obtain the eighth audio data. The first electronic device performs sound effect suppression processing on the audio data of the sixth user according to the second sub-gain attenuation to obtain the ninth audio data; wherein the fifth audio data includes the eighth audio data and the ninth audio data.

[0268] Since the first electronic device can suppress the audio data of different non-target users according to the first sub-gain attenuation and the second sub-gain attenuation, it can further ensure that the audio data of different non-target users have different amplitude changes during the transmission process, so that the user of the first electronic device can hear the voice of the target user clearly and pay attention to the voices of different non-target users.

[0269] Specifically, in the embodiment of the present application, the first electronic device performs sound enhancement processing on the second audio data to obtain fourth audio data, and performs sound suppression processing on the third audio data to obtain fifth audio data. For the specific implementation method, please refer to the specific implementation method in S504 above, which will not be repeated in the embodiment of the present application.

[0270] S1005. The first electronic device outputs target audio data based on the fourth audio data and the fifth audio data.

[0271] After the first electronic device performs sound enhancement processing on the second audio data to obtain fourth audio data, and performs sound suppression processing on the third audio data to obtain fifth audio data, the first electronic device can output target audio data based on the fourth audio data and the fifth audio data.

[0272] In some examples, the first electronic device outputting the target audio data based on the fourth audio data and the fifth audio data may include: the first electronic device accumulates the fourth audio data and the fifth audio data to obtain the target audio data. The first electronic device outputs the target audio data.

[0273] Since the first electronic device can accumulate the fourth audio data and the fifth audio data to obtain the target audio data, when outputting the target audio data, the user of the first electronic device can clearly hear the voice of the target user and can also pay attention to the voices of different users among non-target users.

[0274] Specifically, the specific implementation method of the first electronic device outputting the target audio data based on the fourth audio data and the fifth audio data in the embodiment of the present application can refer to the specific implementation method in the above S505, which will not be repeated in the embodiment of the present application.

[0275] In the solution of the present application, the first electronic device can determine the audio data of the target user (i.e., the main speaker) in the audio data, as well as the audio data of non-target users, based on the audio data sent by the second electronic device. That is, the first electronic device can dynamically determine the audio data of the target user in the audio data sent by the second electronic device. Therefore, the first electronic device can determine the audio data of the target user in real time, so that after the main speaker changes, the first electronic device can still enhance and output the audio data of the changed target user, ensuring that the user can clearly hear the audio data of the changed main speaker. Moreover, in the solution of the present application, since the first electronic device can simultaneously output the audio data of the target user after the enhancement processing and the audio data of the non-target users after the suppression processing, it can ensure that the user can pay attention to the voices of other speakers while hearing the audio data of the main speaker.

[0276] Corresponding to the method in the above-mentioned embodiment, the embodiment of the present application further provides an audio processing device. The audio processing device can be applied to a first electronic device to implement the method in the above-mentioned embodiment. The functions of the audio processing device can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions.

[0277] For example, Fig.11 A schematic diagram of the structure of an audio processing device 11 is shown. Fig.11 As shown, the audio processing device 11 may include: an acquisition module 1101, a determination module 1102, a processing module 1103, and an output module 1104, etc.

[0278] The acquisition module 1101 may be used to acquire call data collected by the second electronic device, where the call data includes first audio data, and the first audio data includes audio data of multiple users.

[0279] The determination module 1102 may be configured to determine audio data of each user among a plurality of users according to the first audio data.

[0280] The determination module 1102 may also be used to determine the second audio data of the target user and the third audio data of the non-target user according to the audio data of each user; the non-target user includes users other than the target user among the multiple users.

[0281] The processing module 1103 can be used to perform sound effect enhancement processing on the second audio data to obtain fourth audio data, and perform sound effect suppression processing on the third audio data to obtain fifth audio data.

[0282] The output module 1104 may be configured to output target audio data based on the fourth audio data and the fifth audio data.

[0283] In another possible implementation, the determination module 1102 may be specifically configured to perform frame processing on the first audio data to obtain a plurality of audio frame data.

[0284] The determination module 1102 may also be specifically configured to perform sound source separation on the audio frame data to obtain sub-audio data of each user in the audio frame data.

[0285] The determination module 1102 may also be specifically configured to obtain the audio data of each user according to the sub-audio data of each user in the plurality of audio frame data.

[0286] In another possible implementation, the determination module 1102 may also be specifically configured to extract voiceprint features from the sub-audio data of the first user to obtain a first voiceprint feature; the first user is any one of the multiple users.

[0287] The determination module 1102 may also be specifically configured to match the first voiceprint feature with a second voiceprint feature in the voiceprint library; the second voiceprint feature is a voiceprint feature of a second user, and the second user is any one of the multiple users.

[0288] The determination module 1102 may also be specifically configured to determine that the first user and the second user are different users when the first voiceprint feature does not match the second voiceprint feature, and determine the sub-audio data of the first user as the audio data of the first user.

[0289] like Fig.11 As shown, the audio processing device 11 may further include a storage module 1105. The storage module 1105 may be used to store the first voiceprint feature in a voiceprint library.

[0290] In another possible implementation, the determination module 1102 may also be specifically used to determine that the first user and the second user are the same user when the first voiceprint feature matches the second voiceprint feature, and determine that the audio data of the second user includes the sub-audio number of the first user.

[0291] In another possible implementation, the determination module 1102 may also be specifically configured to determine the speaking duration and sound energy of each user based on the audio data of each user.

[0292] The determination module 1102 may also be specifically configured to determine a target user according to the speaking time and voice energy of each user; the target user is the user with the longest speaking time and / or the highest voice energy among multiple users.

[0293] The determination module 1102 may also be specifically configured to determine the audio data of the target user as the second audio data, and determine the audio data of the non-target user as the third audio data.

[0294] In another possible implementation, the determination module 1102 may also be specifically configured to determine the lip movement information of each user in the first audio data according to the first image data.

[0295] The determination module 1102 may also be specifically configured to determine the probability of each user speaking based on the lip movement information of each user and the audio data of each user.

[0296] The determination module 1102 may also be specifically configured to determine a target user among multiple users according to the probability of each user speaking.

[0297] The determination module 1102 may also be specifically configured to determine the audio data of the target user as the second audio data, and determine the audio data of the non-target user as the third audio data.

[0298] In another possible implementation, the processing module 1103 may be specifically configured to perform sound enhancement processing on the second audio data according to the transfer function corresponding to the first direction angle and / or the first gain attenuation to obtain fourth audio data.

[0299] The processing module 1103 can also be specifically used to perform sound suppression processing on the third audio data according to the transfer function corresponding to the second direction angle and / or the second gain attenuation to obtain fifth audio data; wherein the first direction angle is smaller than the second direction angle, and / or the first gain attenuation is greater than the second gain attenuation.

[0300] In another possible implementation, the processing module 1103 may be specifically configured to perform sound effect suppression processing on the audio data of the third user according to the transfer function corresponding to the first sub-direction angle to obtain sixth audio data.

[0301] The processing module 1103 can be specifically used to perform sound effect suppression processing on the audio data of the fourth user according to the transfer function corresponding to the second sub-direction angle to obtain seventh audio data; wherein the fifth audio data includes sixth audio data and seventh audio data.

[0302] In another possible implementation, the processing module 1103 may be specifically configured to perform sound effect suppression processing on the audio data of the fifth user according to the first sub-gain attenuation to obtain eighth audio data.

[0303] The processing module 1103 may be specifically configured to perform sound effect suppression processing on the audio data of the sixth user according to the second sub-gain attenuation to obtain ninth audio data; wherein the fifth audio data includes the eighth audio data and the ninth audio data.

[0304] In another possible implementation, the processing module 1103 may also be specifically configured to accumulate the fourth audio data and the fifth audio data to obtain target audio data.

[0305] The output module 1103 may be specifically configured to output target audio data.

[0306] It should be understood that the division of units or modules (hereinafter referred to as units) in the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or physically separated. And the units in the device can all be implemented in the form of software calling through processing elements; or all in the form of hardware; or some units can be implemented in the form of software calling through processing elements, and some units can be implemented in the form of hardware.

[0307] For example, each unit can be a separately established processing element, or it can be integrated in a certain chip of the device. In addition, it can also be stored in a memory in the form of a program, and called and executed by a certain processing element of the device. In addition, all or part of these units can be integrated together, or they can be implemented independently. The processing element described here can also be called a processor, which can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each unit above can be implemented by an integrated logic circuit of hardware in the processor element or in the form of software called by a processing element.

[0308] In one example, the unit in the above apparatus may be one or more integrated circuits configured to implement the above method, such as one or more ASICs, or one or more DSPs, or one or more FPGAs, or a combination of at least two of these integrated circuit forms.

[0309] For another example, when the units in the device can be implemented in the form of a processing element scheduling program, the processing element can be a general-purpose processor, such as a CPU or other processor that can call a program. For another example, these units can be integrated together and implemented in the form of a system on a chip SOC.

[0310] In one implementation, the units of the above apparatus implementing the corresponding steps in the above method can be implemented in the form of a processing element scheduling program. For example, the apparatus may include a processing element and a storage element, and the processing element calls the program stored in the storage element to execute the method described in the above method embodiment. The storage element may be a storage element on the same chip as the processing element, that is, an on-chip storage element.

[0311] In another implementation, the program for executing the above method may be in a storage element on a different chip from the processing element, i.e., an off-chip storage element. In this case, the processing element calls or displays the program on the on-chip storage element from the off-chip storage element to call and execute the method described in the above method embodiment.

[0312] For example, an embodiment of the present application may also provide a device, such as an electronic device, which may include a processor and a memory for storing instructions executable by the processor. When the processor is configured to execute the above instructions, the electronic device implements the audio processing method described in the above embodiment. The memory may be located inside the electronic device or outside the electronic device. And the processor includes one or more.

[0313] In another implementation, the unit of the device implementing each step in the above method may be configured as one or more processing elements, which may be arranged on the corresponding electronic device, and the processing element here may be an integrated circuit, for example: one or more ASICs, or one or more DSPs, or one or more FPGAs, or a combination of these integrated circuits. These integrated circuits may be integrated together to form a chip.

[0314] For example, the present application also provides a chip, which can be applied to the above electronic device. The chip includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through a line; the processor receives and executes computer instructions from the memory of the electronic device through the interface circuit to implement the method described in the above method embodiment.

[0315] An embodiment of the present application also provides a computer program product, including computer instructions executed by the above-mentioned electronic device.

[0316] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0317] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0318] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0319] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0320] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, such as a program. The software product is stored in a program product, such as a computer-readable storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0321] For example, the present application embodiment may also provide a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by an electronic device, the electronic device implements the audio processing method described in the above method embodiment.

[0322] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An audio processing method, characterized in that: Applied to a first electronic device, the method includes: The first electronic device acquires call data collected by the second electronic device, where the call data includes first audio data, and the first audio data includes audio data of multiple users; The first electronic device determines, based on the first audio data, audio data of each user among the multiple users; The first electronic device determines the second audio data of the target user and the third audio data of the non-target user according to the audio data of each user; the non-target user includes users other than the target user among the multiple users; The first electronic device performs sound effect enhancement processing on the second audio data to obtain fourth audio data, and performs sound effect suppression processing on the third audio data to obtain fifth audio data; The first electronic device outputs target audio data based on the fourth audio data and the fifth audio data; The first electronic device performs sound effect enhancement processing on the second audio data to obtain fourth audio data, and performs sound effect suppression processing on the third audio data to obtain fifth audio data, including: The first electronic device performs sound enhancement processing on the second audio data according to the transmission function and / or the first gain attenuation corresponding to the first direction angle to obtain fourth audio data; the first electronic device performs sound suppression processing on the third audio data according to the transmission function and / or the second gain attenuation corresponding to the second direction angle to obtain fifth audio data; the first direction angle is smaller than the second direction angle, and / or the first gain attenuation is greater than the second gain attenuation.

2. The method according to claim 1, characterized in that: The first electronic device determines, according to the first audio data, audio data of each user among the multiple users, including: The first electronic device performs frame processing on the first audio data to obtain a plurality of audio frame data; The first electronic device performs sound source separation on the audio frame data to obtain sub-audio data of each user in the audio frame data; The first electronic device obtains the audio data of each user according to the sub-audio data of each user in the plurality of audio frame data.

3. The method according to claim 2, characterized in that The first electronic device obtains the audio data of each user according to the sub-audio data of each user in the plurality of audio frame data, including: The first electronic device extracts voiceprint features from the sub-audio data of the first user to obtain a first voiceprint feature; the first user is any one of the multiple users; The first electronic device matches the first voiceprint feature with a second voiceprint feature in a voiceprint library; the second voiceprint feature is a voiceprint feature of a second user, and the second user is any one of the multiple users; In the case that the first voiceprint feature does not match the second voiceprint feature, the first electronic device determines that the first user and the second user are different users, and determines the sub-audio data of the first user as the audio data of the first user; The first electronic device stores the first voiceprint feature in the voiceprint library.

4. The method according to claim 3, characterized in that: The method further comprises: When the first voiceprint feature matches the second voiceprint feature, the first electronic device determines that the first user and the second user are the same user, and determines that the audio data of the second user includes the sub-audio number of the first user.

5. The method according to any one of claims 1 to 4, characterized in that The first electronic device determines, according to the audio data of each user, second audio data of a target user and third audio data of a non-target user, including: The first electronic device determines the speaking time and sound energy of each user according to the audio data of each user; The first electronic device determines the target user according to the speaking time and sound energy of each user; the target user is the user with the longest speaking time and / or the highest sound energy among the multiple users; The first electronic device determines the audio data of the target user as the second audio data, and determines the audio data of the non-target user as the third audio data.

6. The method according to any one of claims 1 to 4, characterized in that The call data further includes first image data corresponding to the first audio data, and the first electronic device determines second audio data of a target user and third audio data of a non-target user according to the audio data of each user, including: The first electronic device determines, according to the first image data, lip movement information of each user in the first audio data; The first electronic device determines the probability of each user speaking according to the lip movement information of each user and the audio data of each user; The first electronic device determines the target user from the multiple users according to the probability of each user speaking; The first electronic device determines the audio data of the target user as the second audio data, and determines the audio data of the non-target user as the third audio data.

7. The method according to any one of claims 1 to 4, characterized in that The non-target users include a third user and a fourth user, the third audio data includes audio data of the third user and audio data of the fourth user, the second direction angle includes a first sub-direction angle and a second sub-direction angle, the first sub-direction angle is smaller than the second sub-direction angle, and the first electronic device performs sound effect suppression processing on the third audio data according to a transfer function corresponding to the second direction angle to obtain fifth audio data, including: The first electronic device performs sound effect suppression processing on the audio data of the third user according to the transmission function corresponding to the first sub-direction angle to obtain sixth audio data; The first electronic device performs sound effect suppression processing on the audio data of the fourth user according to the transmission function corresponding to the second sub-direction angle to obtain seventh audio data; The fifth audio data includes the sixth audio data and the seventh audio data.

8. The method according to any one of claims 1 to 4, characterized in that The non-target users include a fifth user and a sixth user, the third audio data includes audio data of the fifth user and audio data of the sixth user, the second gain attenuation includes a first sub-gain attenuation and a second sub-gain attenuation, the first sub-gain attenuation is greater than the second sub-gain attenuation, and the first electronic device performs sound effect suppression processing on the third audio data according to the second gain attenuation to obtain fifth audio data, including: The first electronic device performs sound effect suppression processing on the audio data of the fifth user according to the first sub-gain attenuation to obtain eighth audio data; The first electronic device performs sound effect suppression processing on the audio data of the sixth user according to the second sub-gain attenuation to obtain ninth audio data; The fifth audio data includes the eighth audio data and the ninth audio data.

9. The method according to any one of claims 1 to 4, characterized in that: The first electronic device outputs target audio data based on the fourth audio data and the fifth audio data, including: The first electronic device accumulates the fourth audio data and the fifth audio data to obtain the target audio data; The first electronic device outputs the target audio data.

10. An electronic device, characterized in that: The electronic device comprises a processor and a memory for storing instructions executable by the processor; when the processor is configured to execute the instructions, the electronic device implements the method as claimed in any one of claims 1 to 9.

11. A computer-readable storage medium having computer program instructions stored thereon; characterized in that: When the computer program instructions are executed by an electronic device, the electronic device implements the method according to any one of claims 1 to 9.

Images