Audio processing method, device and electronic equipment

Abstract

The embodiments of the present application disclose an audio processing method, device and electronic device. The electronic device includes a first microphone and a second microphone, the probability of abnormality in the audio data collected by the second microphone is lower than the probability of abnormality in the audio data collected by the first microphone, and the method includes: in response to the audio data collected by the first microphone meeting the abnormal condition, obtaining the collection time of the audio data meeting the abnormal condition as the first collection time; obtaining the first audio data and the second audio data, the first audio data is the audio data related to the first collection time in the audio data collected by the first microphone, and the second audio data is the audio data related to the first collection time in the audio data collected by the second microphone; replacing the first audio data with the second audio data. The above method improves the collection effect of the audio data collected by the first microphone, so as to improve the audio collection effect of the electronic device.

Description

Technical Field

[0001] The present application relates to the field of audio acquisition, and more specifically, to an audio processing method, device and electronic device. Background Art

[0002] When an electronic device is equipped with an audio acquisition device (microphone), recording can be performed through the audio acquisition device. For example, when an electronic device is recording a video, in addition to capturing the video image through a camera, the audio can also be captured synchronously through an audio acquisition device. However, the audio acquisition effect of the relevant electronic devices needs to be improved. Summary of the invention

[0003] In view of the above problems, the present application proposes an audio processing method, device and electronic device to improve the above problems.

[0004] In a first aspect, the present application provides an audio processing method, which is applied to an electronic device, wherein the electronic device includes a first microphone and a second microphone, and the probability of an abnormality in the audio data collected by the second microphone is lower than the probability of an abnormality in the audio data collected by the first microphone. The method includes: in response to the audio data collected by the first microphone satisfying an abnormal condition, obtaining a collection time of the audio data satisfying the abnormal condition as a first collection time; obtaining first audio data and second audio data, the first audio data being audio data related to the first collection time in the audio data collected by the first microphone, and the second audio data being audio data related to the first collection time in the audio data collected by the second microphone; and replacing the first audio data with the second audio data.

[0005] In a second aspect, the present application provides an audio processing device that runs on an electronic device, the electronic device comprising a first microphone and a second microphone, the probability of an abnormality in the audio data collected by the second microphone being lower than the probability of an abnormality in the audio data collected by the first microphone being lower, the device comprising: a time acquisition unit, for obtaining, in response to the audio data collected by the first microphone satisfying an abnormal condition, a collection time of the audio data that meets the abnormal condition as a first collection time; an audio acquisition unit, for obtaining first audio data and second audio data, the first audio data being audio data related to the first collection time in the audio data collected by the first microphone, and the second audio data being audio data related to the first collection time in the audio data collected by the second microphone; and a replacement unit, for replacing the first audio data with the second audio data.

[0006] In a third aspect, the present application provides an electronic device comprising one or more processors and a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.

[0007] In a fourth aspect, the present application provides a computer-readable storage medium, in which a program code is stored, wherein the above method is executed when the program code is run.

[0008] The present application provides an audio processing method, device and electronic device. When the electronic device includes a first microphone and a second microphone, and the probability of the audio data collected by the second microphone being abnormal is lower than the probability of the audio data collected by the first microphone being abnormal, the audio data collected by the first microphone can be responsive to the audio data meeting the abnormal condition, and the collection time of the audio data meeting the abnormal condition can be obtained as the first collection time, and then the audio data related to the first collection time in the audio data collected by the first microphone can be obtained as the first audio data, and the audio data related to the first collection time in the audio data collected by the second microphone can be obtained as the second audio data, and the first audio data can be replaced with the second audio data. Thus, through the above method, the characteristic that the probability of the audio data collected by the second microphone being abnormal is lower than the probability of the audio data collected by the first microphone being abnormal can be utilized. When the audio data collected by the first microphone is abnormal, the audio data collected by the second microphone is used to replace the audio data collected by the first microphone, thereby improving the collection effect of the audio data collected by the first microphone, so as to improve the audio collection effect of the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A schematic diagram showing an application environment of an audio processing method proposed in an embodiment of the present application;

[0011] Figure 2 A schematic diagram showing another application environment of an audio processing method proposed in an embodiment of the present application;

[0012] Figure 3 A flowchart of an audio processing method proposed in an embodiment of the present application is shown;

[0013] Figure 4 A flowchart of an audio data replacement method in an embodiment of the present application is shown;

[0014] Figure 5 A flowchart of an audio processing method proposed in another embodiment of the present application is shown;

[0015] Figure 6 A schematic diagram of first audio data in an embodiment of the present application is shown;

[0016] Figure 7 A schematic diagram showing another first audio data in an embodiment of the present application is shown;

[0017] Figure 8 A flowchart of an audio processing method proposed in another embodiment of the present application is shown;

[0018] Fig. 9 A flowchart showing another audio data replacement method in an embodiment of the present application is shown;

[0019] Fig.10 A flowchart of an audio processing method proposed in another embodiment of the present application is shown;

[0020] Fig.11 A structural block diagram of an audio processing device proposed in an embodiment of the present application is shown;

[0021] Fig.12 It is a storage unit in an embodiment of the present application for storing or carrying a program code for implementing the audio processing method in accordance with an embodiment of the present application;

[0022] Fig.13 A storage unit in real time of the present application for storing or carrying program codes for implementing the audio processing method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] When equipped with a microphone, the electronic device can use the microphone to collect audio in a variety of scenarios. For example, when the electronic device is in a recording scenario, the electronic device can collect audio through the microphone configured therein and store the collected audio data as recording data. Furthermore, in the scenario of video collection, in addition to collecting video images through a camera, the electronic device can also collect ambient sound through a microphone.

[0025] However, the inventors found in the study that the audio collection effect of the relevant electronic equipment needs to be improved. For example, in the relevant electronic equipment, the analog signal collected by the microphone is usually processed by an analog to digital converter (ADC, Analog to Aigital Converter) to convert it into a corresponding digital signal, and then the corresponding audio data is obtained for storage. In some cases, in order to improve the audio collection effect of the microphone (for example, to prevent the audio data collected by the microphone from being broken), two different analog to digital converters can be configured for the microphone. Among them, the analog gains of the two different analog to digital converters are different, the analog gain of one analog to digital converter is higher, and it is used to process signals with smaller amplitudes, and the analog gain of the other analog to digital converter is lower, and it is used to process signals with larger amplitudes. In this case, the electronic device can automatically switch different analog to digital converters according to the current signal strength to avoid distortion of the signal collected by the microphone. However, the number of analog to digital converters in the electronic device is limited. When the number of microphones in the electronic device is large (for example, three), not every microphone can be configured with two analog to digital converters with different analog gains, so that the audio collection effect of the multiple microphones of the electronic device for audio collection together needs to be improved.

[0026] Therefore, the inventors proposed an audio processing method, device and electronic device in the present application. When the electronic device includes a first microphone and a second microphone, and the probability of an abnormality in the audio data collected by the second microphone is lower than the probability of an abnormality in the audio data collected by the first microphone, in response to the audio data collected by the first microphone meeting the abnormal condition, the collection time of the audio data meeting the abnormal condition is obtained as the first collection time, and then the audio data related to the first collection time in the audio data collected by the first microphone is obtained as the first audio data, and the audio data related to the first collection time in the audio data collected by the second microphone is obtained as the second audio data, and the first audio data is replaced with the second audio data.

[0027] Therefore, through the above method, the characteristic that the probability of abnormality in the audio data collected by the second microphone is lower than the probability of abnormality in the audio data collected by the first microphone can be utilized. When the audio data collected by the first microphone is abnormal, the audio data collected by the second microphone is used to replace the audio data collected by the first microphone, thereby improving the collection effect of the audio data collected by the first microphone, thereby improving the audio collection effect of the electronic device.

[0028] An exemplary application scenario involved in an embodiment of the present application is first introduced below.

[0029] like Figure 1 As shown, in Figure 1 The electronic device in the scene shown includes microphone 110, microphone 120, microphone 130 and codec 140. The codec 140 includes a plurality of gain adjustment devices and a plurality of analog-to-digital conversion modules. The microphone 110 and the microphone 130 each may correspond to one analog-to-digital converter, the microphone 110 may correspond to two analog-to-digital converters, and the two analog-to-digital converters corresponding to the microphone 110 each have different analog gains. The analog-to-digital converter corresponding to the microphone can be understood as an analog-to-digital converter for processing the analog signal collected by the microphone. Figure 1 The scenario shown is only exemplary. For example, the number of microphones included in the electronic device may also be as follows: Figure 2 Two shown.

[0030] It should be noted that in the embodiments of the present application, Figure 1 and Figure 2 The microphone 110 shown in the figure may be the first microphone in the embodiment of the present application, and the microphone 120 may be the second microphone in the embodiment of the present application. Figure 1 The microphone 130 in the embodiment of the present application may be the third microphone in the embodiment of the present application. In the embodiment of the present application, the probability of abnormality in the audio data collected by the second microphone is lower than the probability of abnormality in the audio data collected by the first microphone and the third microphone. Among them, the lower probability of abnormality can be understood as the effect of the collected audio data is more stable and less prone to abnormal situations. The abnormal situation may include broken sound or more noise.

[0031] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0032] See also Figure 3 An audio processing method provided in an embodiment of the present application is applied to an electronic device, wherein the electronic device includes a first microphone and a second microphone, and the probability that the audio data collected by the second microphone is abnormal is lower than the probability that the audio data collected by the first microphone is abnormal, and the method includes:

[0033] S110: In response to the audio data collected by the first microphone satisfying an abnormal condition, obtaining a collection time of the audio data satisfying the abnormal condition as a first collection time.

[0034] In the embodiment of the present application, the first microphone can collect audio in a variety of situations. As one way, the electronic device can start the recording function in response to the recording start instruction, and when the recording function is started, the first microphone will be controlled to collect audio. As another way, the electronic device can start the image acquisition function in response to the image acquisition instruction, and when the image acquisition function is started, in addition to controlling the camera to collect video images, the first microphone will also be controlled to collect audio.

[0035] Furthermore, in the embodiment of the present application, when the first microphone starts to collect audio, the second microphone may also start to collect audio synchronously.

[0036] Among them, the abnormal condition is a condition for triggering the replacement of audio data. And, in an embodiment of the present application, the abnormal condition may have multiple meanings. Optionally, the abnormal condition may represent that the audio data currently being detected is abnormal. Furthermore, optionally, the abnormal condition may represent that the audio data collected after the audio data currently being detected may be abnormal. Then, in the case where there are multiple meanings represented by the abnormal condition and the multiple meanings are different, there will also be multiple corresponding ways to replace the audio data (which will be introduced in detail in subsequent embodiments).

[0037] In the embodiments of the present application, the abnormality of the audio data can also correspond to multiple understandings. As one way, it can be determined that the audio data is abnormal when the audio data is detected to be broken. As another way, it can be determined that the audio data is abnormal when it is detected that the audio data includes a specified sound. The specified sound may include a specified type of noise. For example, wind noise.

[0038] In the case where audio data may be determined to be abnormal in various situations, the manner of determining whether the audio data meets the abnormal condition may also be different accordingly.

[0039] Optionally, when the audio data is abnormal in that the audio data is broken, as a method, if it is detected that the amplitude of the audio data collected by the first microphone is greater than the preset amplitude, it is determined that the corresponding audio data with an amplitude greater than the preset amplitude meets the abnormal condition. Optionally, when the audio data is abnormal in that the audio data includes a specified sound, if it is detected that the audio data collected by the first microphone includes the specified sound, it is determined that the audio data including the specified sound meets the abnormal condition.

[0040] It should be noted that, in the process of synchronously collecting audio by the first microphone and the second microphone, the audio data collected at the same collection time are interrelated. For example, when there is no abnormality in the audio data collected by the first microphone and the second microphone, the audio data collected by the first microphone and the second microphone at the same collection time may play the same content. In this case, if the audio data collected by the first microphone needs to be replaced, the first collection time can be obtained first, so that the audio data that needs to be replaced in the audio data collected by the first microphone can be determined according to the first collection time, and the audio data for replacement in the audio data collected by the second microphone can also be determined according to the first collection time.

[0041] In the embodiment of the present application, the first acquisition time can be understood as a sampling moment, for example, an audio sampling point. It can also be understood as a time period consisting of multiple sampling moments. In the case of a time period, the first acquisition time will include multiple continuous audio sampling points.

[0042] S120: Acquire first audio data and second audio data, where the first audio data is audio data associated with a first acquisition time in the audio data collected by the first microphone, and the second audio data is audio data associated with the first acquisition time in the audio data collected by the second microphone.

[0043] After obtaining the first acquisition time, the first audio data and the second audio data can be further acquired, wherein the first audio data is the audio data to be replaced in the audio data acquired by the first microphone, and the second audio parameter can be understood as the audio data used to replace the first audio data.

[0044] S130: Replace the first audio data with the second audio data.

[0045] It should be noted that, in the process of collecting audio data, the electronic device will store the audio data collected by the first microphone as recording data, or as sound data in the video data. Replacing the first audio data with the second audio data can be understood as replacing the first audio data with audio data that is closer to the second audio data, so that the audio data finally stored by the electronic device will be replaced by the first audio data with the audio data that is closer to the second audio data. Closer to the second audio data can be understood as the audio parameters of the audio data are closer to the audio parameters of the second audio data.

[0046] As an alternative, the first audio data may be directly replaced with the second audio data, so that the electronic device will eventually store the second audio data and discard the first audio data. Furthermore, as another alternative, the electronic device may first generate corresponding replacement audio data according to the second audio data, and then replace the first audio data with the replacement audio data.

[0047] For example, Figure 4 As shown, the audio data collected by the first microphone includes audio data A1, audio data A2, audio data A3 and audio data A4. The audio data collected by the second microphone includes audio data B1, audio data B2, audio data B3 and audio data B4. Among them, if when detecting the audio data A2 collected by the first microphone, it is detected that the audio data A2 is abnormal, then the collection time of the audio data A2 is determined to be the first collection time, and then the audio data A2 will be used as the first audio data. Correspondingly, in the audio data collected by the second microphone, the corresponding audio data with the collection time of the first collection time is audio data B2. If Figure 4 In the example shown, the abnormal condition indicates that the audio data being detected is abnormal, so audio data B2 will be used to replace audio data A2, and the final stored audio data (i.e., the audio data subsequently used for playback) includes audio data A1, audio data B2, audio data A3 and audio data A4.

[0048] It should be noted that the audio processing method provided in the embodiment of the present application can be applied to the process of real-time audio collection by microphones (first microphone and second microphone). In this case, as a method, in the process of audio collection by the electronic device, the electronic device can respond to the audio data collected by the first microphone meeting the abnormal condition, and obtain the collection time of the audio data meeting the abnormal condition as the first collection time. It can be understood that in this way, if the electronic device detects that the currently collected audio data (first audio data) meets the abnormal condition, and the abnormal condition characterizes that the audio data being detected is abnormal, the second audio data will be used to directly replace the currently collected audio data.

[0049] Furthermore, the audio processing method provided in the embodiment of the present application can be applied after the microphones (the first microphone and the second microphone) complete audio collection. In this case, as a method, in the process of audio detection performed after the electronic device completes audio collection, the electronic device can respond to the audio data collected by the first microphone meeting the abnormal condition, and obtain the collection time of the audio data that meets the abnormal condition as the first collection time.

[0050] An audio processing method provided by the present embodiment can, in response to the audio data collected by the first microphone satisfying an abnormal condition, obtain the collection time of the audio data satisfying the abnormal condition as the first collection time, further obtain the audio data corresponding to the first collection time in the audio data collected by the first microphone as the first audio data, obtain the audio data corresponding to the first collection time in the audio data collected by the second microphone as the second audio data, and replace the first audio data with the second audio data, in a case where the electronic device includes a first microphone and a second microphone and the probability of the audio data collected by the first microphone being abnormal is higher than the probability of the audio data collected by the second microphone being abnormal.

[0051] Therefore, through the above method, the characteristic that the probability of abnormality in the audio data collected by the first microphone is higher than the probability of abnormality in the audio data collected by the second microphone can be utilized. When the audio data collected by the first microphone is abnormal, the audio data collected by the second microphone is used to replace the audio data collected by the first microphone, thereby improving the collection effect of the audio data collected by the first microphone, thereby improving the audio collection effect of the electronic device.

[0052] See also Figure 5 An audio processing method provided in an embodiment of the present application is applied to an electronic device, the electronic device includes a first microphone and a second microphone, the probability that the audio data collected by the first microphone is abnormal is higher than the probability that the audio data collected by the second microphone is abnormal, and the method includes:

[0053] S310: In response to the audio data collected by the first microphone satisfying an abnormal condition, obtaining a collection time of the audio data satisfying the abnormal condition as a first collection time.

[0054] S320: Acquire audio data collected by the first microphone at a collection time after the first collection time as first audio data.

[0055] In this embodiment, the abnormal condition may indicate that the audio data collected after the currently detected audio data may be abnormal. In this case, when it is determined that the currently collected audio data meets the abnormal condition, the collection time of the currently collected audio data can be used as the first collection time, and correspondingly, the audio data collected by the first microphone after the first collection time can be used as the first audio data.

[0056] S330: Acquire audio data collected by the second microphone at a collection time after the first collection time as second audio data.

[0057] After the first collection time is determined, the second audio data is also obtained through the audio data collected by the second microphone.

[0058] S340: Replace the first audio data with the second audio data.

[0059] In one embodiment, after the audio data collected by the first microphone at the first collection time is used as the first audio data and it is determined that the first audio data collected by the first microphone is replaced with the second audio data collected by the second microphone, the audio data collected by the first microphone after the first collection time may also be replaced. For example, Figure 6 As shown, if the audio data A2 collected by the first microphone is detected and it is determined that the audio data A2 meets the abnormal condition, the collection time of the audio data A2 will be used as the first collection time, and accordingly, the audio data A2 will be used as the first audio data. Correspondingly, the audio data A3, audio data A4, audio data A5 and audio data A6 collected after the first collection time will also be used as the first audio data.

[0060] In another way, after determining that the audio data collected by the first microphone needs to be replaced, the audio data collected by the first microphone after the first collection time may continue to be detected. If it is detected that the audio data does not meet the abnormal condition, the collection time of the audio data that does not meet the abnormal condition is used as the second collection time, and the audio data collected by the first microphone will not be replaced after the second collection time. Figure 7 As shown, if when detecting the audio data A2 collected by the first microphone, it is determined that the audio data A2 meets the abnormal condition, the collection time of the audio data A2 will be used as the first collection time. For the audio data collected after the audio data A2, detection will continue. And, when it is detected that the audio data A5 does not meet the abnormal condition, the collection time of the audio data A5 will be used as the second collection time. Correspondingly, the audio data A3 and the audio data A4 collected between the first collection time and the second collection time are used as the first audio data. It should be noted that when each audio data is detected, the first collection time can be understood as the collection time of the audio data that meets the abnormal condition for the first time when it is not determined that the audio data needs to be replaced.

[0061] like Figure 7 As shown in the content, the first audio data includes audio data collected at multiple collection times. In this case, the first audio data can be understood as including multiple first sub-audio data, wherein each first sub-audio data can be understood as audio data collected at a collection time between the first collection time and the second collection time. For example, Figure 7Taking the content shown in FIG. 1 as an example, when the first audio data includes audio data A3 and audio data A4, audio data A3 and audio data A4 can both be understood as a first sub audio data.

[0062] Correspondingly, the second audio data includes a plurality of second sub-audio data collected by the second microphone between the first collection time and the second collection time, and the second collection time is the time when the audio data collected by the first microphone is first detected to be normal after the first collection time. In this way, replacing the first audio data with the second audio data may include: replacing the plurality of sub-first audio data with the second sub-audio data with the same collection time as the corresponding ones.

[0063] As a method, the setting positions of the first microphone and the second microphone may be different. In this case, even if the audio data collected by the first microphone and the second microphone at the same time, the content played when playing may be different. In this case, if the first audio data is directly replaced with the second audio data, there may be a sudden change in sound when the audio data collected by the first microphone is played. In order to improve the situation where the sound suddenly changes after the audio data is replaced, the first sub-audio data included in the first audio data may be gradually replaced. In this case, the time between the first acquisition time and the second acquisition time may be divided into multiple first acquisition time periods and at least one second acquisition time period, and at least one second acquisition time period is after the first acquisition time period. In this case, if Figure 8 As shown, based on a plurality of second sub audio data, the first sub audio data corresponding to each other and having the same acquisition time are replaced, including:

[0064] S341: According to the weights of the first sub audio data corresponding to each of the multiple first acquisition time periods, and the weights of the second sub audio data corresponding to each of the multiple first acquisition time periods, replacement audio data corresponding to each of the multiple first acquisition time periods are obtained, wherein the sum of the weights of the first sub audio data and the second sub audio data having the same corresponding first acquisition time period is 1, and as the first acquisition time period increases, the weight corresponding to the corresponding first sub audio data becomes smaller.

[0065] As a method, according to the weights of the first sub-audio data corresponding to each of the multiple first acquisition time periods and the weights of the second sub-audio data corresponding to each of the multiple first acquisition time periods, the replacement audio data corresponding to each of the multiple first acquisition time periods is obtained, including: multiplying the amplitude of the first sub-audio data corresponding to the first acquisition time period by the corresponding weight to obtain a first intermediate amplitude; multiplying the amplitude of the second sub-audio data corresponding to the first acquisition time period by the corresponding weight to obtain a second intermediate amplitude; obtaining the sum of the first intermediate amplitude and the second intermediate amplitude as a first reference amplitude. Based on the first reference amplitude, the corresponding replacement audio data is obtained.

[0066] For example, Fig. 9 As shown, the first sub audio data corresponding to the first collection time period in the audio data collected by the first microphone includes audio data A3, audio data A4, audio data A5 and audio data A6. The first sub audio data corresponding to the first collection time period includes audio data A7.

[0067] Among them, w1+w2=1, w3+w4=1, w5+w6=1, w7+w8=1. Moreover, w7 is greater than w5, w5 is greater than w3, and w3 is greater than w1. Among them, for the audio data A3 and the audio data B3 both corresponding to the first acquisition time period t3, the replacement audio data C3 corresponding to the first acquisition time period t3 is obtained by the weight w1 corresponding to the audio data A3 and the weight w2 corresponding to the audio data B3. Similarly, the replacement audio data C4 corresponding to the first acquisition time period t4, the replacement audio data C5 corresponding to the first acquisition time period t5, and the replacement audio data C6 corresponding to the first acquisition time period t6 can be obtained.

[0068] According to the content of the aforementioned embodiment, it can be understood that as the first acquisition time period increases, the weight corresponding to the first sub-audio data becomes smaller and smaller, so that the content about the first sub-audio data in the replacement audio data corresponding to a larger first acquisition time period becomes less and less, and the content about the second sub-audio data becomes more and more, so that the first sub-audio data can be gradually replaced with the corresponding second sub-audio data to avoid sudden changes in sound when the audio data finally collected by the first microphone is played.

[0069] S342: Replace the first sub audio data corresponding to each of the plurality of first acquisition time periods with the corresponding replacement audio data.

[0070] S343: Replace the first sub audio data corresponding to at least one second acquisition time period with the corresponding second sub audio data.

[0071] like Fig. 9As shown, the first sub audio data corresponding to the second acquisition time period includes audio data A7, and the audio data A7 will be directly replaced by audio data B7.

[0072] It should be noted that, in one approach, the audio data collected corresponding to the first collection time may also be used as the first sub-audio data. In this approach, the first audio data may be understood to include audio data that meets the abnormal condition and audio data collected between the first collection time and the second collection time.

[0073] An audio processing method provided in this embodiment can utilize the characteristic that the probability of anomalies in audio data collected by a first microphone is higher than the probability of anomalies in audio data collected by a second microphone. When an abnormal condition indicates that the audio data collected by the first microphone may be abnormal, when the first audio data collected by the first microphone after a first collection time is abnormal, the first audio data is replaced by the second audio data collected by the second microphone after the first collection time, thereby timely reducing the probability of anomalies in the audio data collected by the first microphone, thereby improving the audio collection effect of the electronic device.

[0074] See also Fig.10 An audio processing method provided in an embodiment of the present application is applied to an electronic device, the electronic device includes a first microphone, a second microphone and a third microphone, the probability that the audio data collected by the first microphone is abnormal is higher than the probability that the audio data collected by the second microphone is abnormal, and the method includes:

[0075] S410: In response to the audio data collected by the first microphone satisfying an abnormal condition, obtaining a collection time of the audio data satisfying the abnormal condition as a first collection time.

[0076] S420: Acquire first audio data and second audio data, where the first audio data is audio data associated with a first acquisition time in the audio data collected by the first microphone, and the second audio data is audio data associated with the first acquisition time in the audio data collected by the second microphone.

[0077] S430: Replace the first audio data and the third audio data with the second audio data, where the third audio data is the audio data collected by the third microphone and corresponds to the first collection time.

[0078] As a method, the first microphone can be the main microphone of the electronic device, the third microphone can be the secondary microphone of the electronic device, and the second microphone can be the camera microphone of the electronic device. In this way, during the audio collection process, the phases of the audio data collected by the first microphone and the second microphone will be different, so that when the audio data collected by the first microphone and the second microphone are played, there will be a better sense of stereo. In order to further improve the audio collection effect, the amplitude of the first audio data can be replaced with the amplitude of the second audio data, and the phase of the first audio data is retained. In this way, the audio data collected by the first microphone can be prevented from being broken during playback, and the original phase of the audio data collected by the first microphone can be retained to retain the sense of stereo during playback.

[0079] Among them, the electronic device also includes a first analog-to-digital converter, a second analog-to-digital converter, a third analog-to-digital converter and a fourth analog-to-digital converter; the first analog-to-digital converter is used to process the analog signal collected by the first microphone, the second analog-to-digital converter is used to process the analog signal collected by the third microphone, the third analog-to-digital converter and the fourth analog-to-digital converter are used to collaboratively process the analog signal collected by the second microphone, and the analog gains of the third analog-to-digital converter and the fourth analog-to-digital converter are different.

[0080] In the case where the electronic device can replace only the first audio data or can replace both the first audio data and the third audio data, the electronic device can determine a specific replacement method according to current actual conditions.

[0081] As a method, a specific replacement method can be configured by a user. In this method, the electronic device can be configured with a replacement method selection interface, in which a first method option and a second method option are configured. Among them, the method corresponding to the first method option is to replace only the first audio data, and the method corresponding to the second method option is to replace both the first audio data and the third audio data. After the user determines the required replacement method through the replacement method selection interface, the electronic device will store the method determined by the user. When audio data needs to be replaced, the electronic device can replace the audio data according to the stored replacement method.

[0082] As another way, the electronic device can determine the corresponding replacement method according to the type of the collected audio. Among them, if the electronic device detects that the collected audio is of a music type, the determined replacement method is to replace only the first audio data. Thereby, when the audio is collected synchronously by the first microphone and the third microphone, although part of the audio data collected by the first microphone is replaced by the audio data collected by the second microphone, the phase of the audio data collected by the second microphone is different from the phase of the audio data collected by the third microphone, so that the final collected audio data still has a three-dimensional sense when playing. Correspondingly, if the electronic device detects that the collected audio is of a type other than music, both the first audio data and the third audio data can be replaced.

[0083] The present embodiment provides an audio processing method, so that through the above-mentioned method, the characteristic that the probability of abnormality of the audio data collected by the first microphone is higher than the probability of abnormality of the audio data collected by the second microphone can be utilized. When the audio data collected by the first microphone is abnormal, the audio data collected by the second microphone is used to replace the audio data collected by the first microphone, thereby improving the collection effect of the audio data collected by the first microphone, so as to improve the audio collection effect of the electronic device. Moreover, in the present embodiment, when the audio data is abnormal, including the audio data being broken, the audio data collected by the first microphone and the third microphone are replaced by the audio data collected by the second microphone, so that when not every microphone corresponds to two analog-to-digital converters with different analog gains, the microphone corresponding to only one analog-to-digital converter can also have a better audio collection effect, thereby making the electronic device have a better anti-breaking effect in the process of audio collection, thereby improving the overall audio collection effect of the electronic device.

[0084] See also Fig.11 An audio processing device 400 provided in an embodiment of the present application runs on an electronic device, the electronic device includes a first microphone and a second microphone, the probability that the audio data collected by the first microphone is abnormal is higher than the probability that the audio data collected by the second microphone is abnormal, and the device 400 includes:

[0085] The time acquisition unit 410 is configured to acquire, in response to the audio data collected by the first microphone satisfying an abnormal condition, a collection time of the audio data satisfying the abnormal condition as a first collection time.

[0086] As a method, the time acquisition unit 410 is also used to determine that the corresponding audio data with an amplitude greater than the preset amplitude meets the abnormal condition if it is detected that the amplitude of the audio data collected by the first microphone is greater than the preset amplitude.

[0087] As a method, the time acquisition unit 410 is specifically used to, in a process of audio collection by the electronic device, in response to the audio data collected by the first microphone satisfying the abnormal condition, obtain the collection time of the audio data that meets the abnormal condition as the first collection time; or, in a process of audio detection performed after the electronic device completes audio collection, in response to the audio data collected by the first microphone satisfying the abnormal condition, obtain the collection time of the audio data that meets the abnormal condition as the first collection time.

[0088] The audio acquisition unit 420 is used to acquire first audio data and second audio data, where the first audio data is audio data related to the first acquisition time in the audio data collected by the first microphone, and the second audio data is audio data related to the first acquisition time in the audio data collected by the second microphone.

[0089] The replacing unit 430 is configured to replace the first audio data with the second audio data.

[0090] As a method, the audio acquisition unit 420 is specifically used to acquire the audio data collected by the first microphone at the first collection time as the first audio data, and acquire the audio data collected by the second microphone at the first collection time as the second audio data. In this method, the replacement unit 430 is specifically used to replace the first audio data with the second audio data.

[0091] As a method, the audio acquisition unit 420 is specifically used to acquire the audio data collected by the first microphone at a collection time after the first collection time as the first audio data, and to acquire the audio data collected by the second microphone at a collection time after the first collection time as the second audio data. Optionally, the first audio data includes a plurality of first sub-audio data collected by the first microphone between the first collection time and the second collection time, and the second audio data includes a plurality of second sub-audio data collected by the second microphone between the first collection time and the second collection time, and the second collection time is the time when the audio data collected by the first microphone is first detected to be normal after the first collection time. Correspondingly, the replacement unit 430 is specifically used to replace the first sub-audio data with the same corresponding collection time based on the plurality of second sub-audio data.

[0092] As a mode, the first acquisition time and the second acquisition time include multiple first acquisition time periods and at least one second acquisition time period, and at least one second acquisition time period is after the first acquisition time period. The replacement unit 430 is specifically used to obtain replacement audio data corresponding to each of the multiple first acquisition time periods according to the weights of the first sub-audio data corresponding to each of the multiple first acquisition time periods and the weights of the second sub-audio data corresponding to each of the multiple first acquisition time periods, wherein the sum of the weights of the first sub-audio data and the second sub-audio data having the same corresponding first acquisition time period is 1, and as the first acquisition time period increases, the corresponding weight of the corresponding first sub-audio data decreases; replace the first sub-audio data corresponding to each of the multiple first acquisition time periods with the corresponding replacement audio data; replace the first sub-audio data corresponding to each of the multiple second acquisition time periods with the corresponding second sub-audio data.

[0093] Optionally, the replacement unit 430 is specifically used to multiply the amplitude of the first sub-audio data corresponding to the first acquisition time period by the corresponding weight to obtain a first intermediate amplitude; multiply the amplitude of the second sub-audio data corresponding to the first acquisition time period by the corresponding weight to obtain a second intermediate amplitude; obtain the sum of the first intermediate amplitude and the second intermediate amplitude as the first reference amplitude; and obtain corresponding replacement audio data based on the first reference amplitude.

[0094] As a mode, the electronic device also includes a third microphone. In this mode, the replacement unit 430 is specifically used to replace the first audio data and the third audio data according to the second audio data, wherein the third audio data is the audio data corresponding to the first acquisition time in the audio data collected by the third microphone. Optionally, the electronic device also includes a first analog-to-digital converter, a second analog-to-digital converter, a third analog-to-digital converter and a fourth analog-to-digital converter; the first analog-to-digital converter is used to process the analog signal collected by the first microphone, the second analog-to-digital converter is used to process the analog signal collected by the third microphone, the third analog-to-digital converter and the fourth analog-to-digital converter are used to collaboratively process the analog signal collected by the second microphone, and the analog gains of the third analog-to-digital converter and the fourth analog-to-digital converter are different.

[0095] As a way, the replacement unit 430 is specifically configured to replace the amplitude of the first audio data with the amplitude of the second audio data, and retain the phase of the first audio data.

[0096] The present embodiment provides an audio processing device, through which the characteristic that the probability of anomalies in audio data collected by a first microphone is higher than the probability of anomalies in audio data collected by a second microphone can be utilized. When an abnormality occurs in the audio data collected by the first microphone, the audio data collected by the second microphone is used to replace the audio data collected by the first microphone, thereby improving the collection effect of the audio data collected by the first microphone, thereby improving the audio collection effect of the electronic device.

[0097] It should be noted that the device embodiment in the present application corresponds to the aforementioned method embodiment. The specific principles in the device embodiment can be found in the contents of the aforementioned method embodiment and will not be repeated here.

[0098] See also Fig.12 Based on the above-mentioned audio processing method and device, the embodiment of the present application also provides another electronic device 200 that can execute the above-mentioned audio processing method. The electronic device 200 includes one or more (only one is shown in the figure) processors 202, a memory 204, a network module 206, a sensor module 208 and an audio acquisition device 210 that are coupled to each other. Among them, the memory 204 stores a program that can execute the content of the above-mentioned embodiment, and the processor 202 can execute the program stored in the memory 204.

[0099] Among them, the processor 202 may include one or more processing cores. The processor 202 uses various interfaces and lines to connect various parts within the entire electronic device 200, and executes various functions and processes data of the electronic device 200 by running or executing instructions, programs, code sets or instruction sets stored in the memory 204, and calling data stored in the memory 204. Optionally, the processor 202 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 202 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 202, but may be implemented separately through a communication chip.

[0100] The memory 204 may include a random access memory (RAM) or a read-only memory (ROM). The memory 204 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 204 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc.

[0101] The network module 206 is used to implement information exchange between the electronic device 200 and other devices, for example, transmission of device control instructions, operation request instructions, and status information acquisition instructions, etc. When the electronic device 200 is specifically a different device, the corresponding network module 206 may be different.

[0102] The sensor module 208 may include at least one sensor. Specifically, the sensor module 208 may include but is not limited to: a light sensor, a motion sensor, a pressure sensor, an infrared heat sensor, a distance sensor, an acceleration sensor, and other sensors.

[0103] Among them, the pressure sensor can be a sensor that detects the pressure generated by pressing on the electronic device 200. That is, the pressure sensor detects the pressure generated by the contact or pressing between the user and the electronic device, such as the pressure generated by the contact or pressing between the user's ear and the mobile terminal. Therefore, the pressure sensor can be used to determine whether contact or pressing occurs between the user and the electronic device 200, and the magnitude of the pressure.

[0104] Among them, the acceleration sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary, and can be used for applications that recognize the posture of the electronic device 200 (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. In addition, the electronic device 200 can also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, etc., which will not be repeated here.

[0105] The audio acquisition device 210 is used to collect audio signals. Optionally, the audio acquisition device 210 includes a plurality of audio acquisition devices. The audio acquisition device may be a microphone. For example, in one embodiment, the audio acquisition device 210 may include two microphones, in which one of the microphones may correspond to an analog-to-digital converter, and the other microphone may correspond to an analog-to-digital converter with two different analog gains. In another embodiment, the audio acquisition device 210 may include three microphones. In this embodiment, two of the microphones (e.g., a main microphone and a secondary microphone) may each correspond to an analog-to-digital converter, and the other microphone (e.g., a camera microphone) may correspond to an analog-to-digital converter with two different analog gains.

[0106] As a way, the network module of the electronic device 200 is a radio frequency module, which is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The radio frequency module may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity modules (SIM) cards, memories, etc. For example, the radio frequency module can exchange information with an external device through the electromagnetic waves sent or received, and then receive audio signals sent by the external device.

[0107] Furthermore, the electronic device 200 may also include an image acquisition device for image acquisition, for example, a video, a static picture or a dynamic picture may be taken by the image acquisition device.

[0108] Please refer to Fig.13 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 900 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0109] The computer-readable storage medium 900 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium 900 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 900 has storage space for program code 910 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 910 can be compressed, for example, in an appropriate form.

[0110] In summary, the present application provides an audio processing method, device, and electronic device. When the electronic device includes a first microphone and a second microphone, and the probability of an abnormality in the audio data collected by the second microphone is lower than the probability of an abnormality in the audio data collected by the first microphone, the audio data collected by the first microphone can be responsive to the audio data collected by the first microphone meeting the abnormal condition, and the collection time of the audio data meeting the abnormal condition can be obtained as the first collection time, and then the audio data related to the first collection time in the audio data collected by the first microphone can be obtained as the first audio data, and the audio data related to the first collection time in the audio data collected by the second microphone can be obtained as the second audio data, and the first audio data can be replaced with the second audio data. Thus, through the above method, the characteristic that the probability of an abnormality in the audio data collected by the second microphone is lower than the probability of an abnormality in the audio data collected by the first microphone can be utilized. When the audio data collected by the first microphone is abnormal, the audio data collected by the second microphone is used to replace the audio data collected by the first microphone, thereby improving the collection effect of the audio data collected by the first microphone, so as to improve the audio collection effect of the electronic device.

[0111] Therefore, through the above method, in addition to collecting audio signals through the audio collection device of the electronic device itself, audio signals can also be collected with the help of the audio collection device of an external device, so that the electronic device can collect more effective audio signals with the help of the external device. After determining the recording direction, the target audio signal corresponding to the recording direction in the audio signal (third audio signal) collected jointly by the electronic device and the external device can be highlighted, so as to improve the sound clarity of the specific object.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An audio processing method, characterized in that: Applied to an electronic device, the electronic device includes a first microphone and a second microphone, the probability of abnormality of audio data collected by the second microphone is lower than the probability of abnormality of audio data collected by the first microphone, and the method includes: In response to the audio data collected by the first microphone meeting an abnormal condition, obtaining a collection time of the audio data meeting the abnormal condition as a first collection time, where the first collection time is a sampling moment, or the first collection time is a time period including multiple sampling moments; Acquire first audio data and second audio data, wherein the first audio data is audio data associated with the first acquisition time among the audio data collected by the first microphone, and the second audio data is audio data associated with the first acquisition time among the audio data collected by the second microphone; The first audio data is replaced with the second audio data, wherein replacing the first audio data with the second audio data comprises: replacing the amplitude of the first audio data with the amplitude of the second audio data, and retaining the phase of the first audio data.

2. The method according to claim 1, characterized in that The obtaining of the first audio data and the second audio data includes: The audio data collected by the first microphone at the first collection time is obtained as the first audio data, and the audio data collected by the second microphone at the first collection time is obtained as the second audio data.

3. The method according to claim 1, characterized in that The first audio data includes a plurality of first sub-audio data collected by the first microphone between the first collection time and a second collection time, and the second audio data includes a plurality of second sub-audio data collected by the second microphone between the first collection time and the second collection time, and the second collection time is the time when the audio data collected by the first microphone is first detected to be normal after the first collection time; The replacing the first audio data with the second audio data comprises: The multiple first sub-audio data are replaced by the second sub-audio data corresponding to each other and having the same collection time.

4. The method according to claim 3, characterized in that The first acquisition time and the second acquisition time include a plurality of first acquisition time periods and at least one second acquisition time period, wherein the at least one second acquisition time period is after the first acquisition time period, and the replacing the plurality of first sub-audio data with second sub-audio data having the same corresponding acquisition time includes: According to the weights of the first sub audio data corresponding to each of the plurality of first collection time periods and the weights of the second sub audio data corresponding to each of the plurality of first collection time periods, the replacement audio data corresponding to each of the plurality of first collection time periods are obtained, wherein the sum of the weights of the first sub audio data and the second sub audio data having the same first collection time period is 1, and as the first collection time period increases, the weight corresponding to the corresponding first sub audio data decreases; Replacing the first sub audio data corresponding to each of the plurality of first acquisition time periods with the corresponding replacement audio data; The first sub audio data corresponding to at least one of the second acquisition time periods is replaced with the corresponding second sub audio data.

5. The method according to claim 4, characterized in that The obtaining, according to the weights of the first sub-audio data corresponding to the plurality of first collection time periods, and the weights of the second sub-audio data corresponding to the plurality of first collection time periods, the replacement audio data corresponding to each of the plurality of first collection time periods comprises: multiplying the amplitude of the first sub-audio data corresponding to the first acquisition time period by the corresponding weight to obtain a first intermediate amplitude; multiplying the amplitude of the second sub-audio data corresponding to the first acquisition time period by the corresponding weight to obtain a second intermediate amplitude; Acquire the sum of the first intermediate amplitude and the second intermediate amplitude as a first reference amplitude; Corresponding replacement audio data is obtained based on the first reference amplitude.

6. The method according to any one of claims 1 to 5, characterized in that: The electronic device further includes a third microphone, the probability that the audio data collected by the second microphone is abnormal is lower than the probability that the audio data collected by the third microphone is abnormal, the setting position of the third microphone is different from the setting position of the first microphone, and the replacing the first audio data with the second audio data includes: The first audio data and the third audio data are replaced by the second audio data, wherein: The third audio data is audio data corresponding to the first collection time among the audio data collected by the third microphone.

7. The method according to claim 6, characterized in that The electronic device also includes a first analog-to-digital converter, a second analog-to-digital converter, a third analog-to-digital converter and a fourth analog-to-digital converter; the first analog-to-digital converter is used to process the analog signal collected by the first microphone, the second analog-to-digital converter is used to process the analog signal collected by the third microphone, the third analog-to-digital converter and the fourth analog-to-digital converter are used to collaboratively process the analog signal collected by the second microphone, and the third analog-to-digital converter and the fourth analog-to-digital converter have different analog gains.

8. An audio processing device, characterized in that: The device is operated on an electronic device, the electronic device includes a first microphone and a second microphone, the probability that the audio data collected by the second microphone is abnormal is lower than the probability that the audio data collected by the first microphone is abnormal, and the device includes: a time acquisition unit, configured to, in response to the audio data collected by the first microphone satisfying an abnormal condition, acquire a collection time of the audio data satisfying the abnormal condition as a first collection time, wherein the first collection time is a sampling moment, or the first collection time is a time period including a plurality of sampling moments; an audio acquisition unit, configured to acquire first audio data and second audio data, wherein the first audio data is audio data associated with the first acquisition time among the audio data collected by the first microphone, and the second audio data is audio data associated with the first acquisition time among the audio data collected by the second microphone; A replacing unit is used to replace the amplitude of the first audio data with the amplitude of the second audio data, and retain the phase of the first audio data.

9. An electronic device, characterized in that: comprising a first microphone and a second microphone, one or more processors and a memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, wherein when the program code is run, the method according to any one of claims 1 to 7 is executed.

Images