Audio processing method and device, electronic equipment and medium

Abstract

The invention relates to an audio processing method and device, electronic equipment and a medium. The audio processing method comprises the following steps: determining to-be-played first audio data in target audio data; under the condition that invalid data exist in the first audio data, generating second audio data according to at least part of audio data except the invalid data in the target audio data; and replacing invalid data in the first audio data with the second audio data. The effective second audio data is generated to replace the invalid data, so that noise caused by playing the invalid data is avoided, and the noise in the audio playing process is eliminated.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of audio technology, and in particular, to an audio processing method and device, electronic equipment and medium. BACKGROUND

[0002] With the development of electronic equipment, electronic equipment can not only be used for communication, but also can be used for playing high-quality audio. However, due to the limited performance of electronic equipment or design defects of the application program generating the audio, there is invalid data in the audio, which causes noise in the process of playing the audio. SUMMARY

[0003] To overcome the problems in the related art, the present disclosure provides an audio processing method, device, electronic equipment and medium.

[0004] According to a first aspect of an embodiment of the present disclosure, an audio processing method is provided, the audio processing method comprising:

[0005] determining first audio data to be played in target audio data;

[0006] in a case where there is invalid data in the first audio data, generating second audio data according to at least part of the audio data in the target audio data except the invalid data;

[0007] replacing the invalid data in the first audio data with the second audio data.

[0008] In some embodiments of the present disclosure, the generating second audio data according to at least part of the audio data in the target audio data except the invalid data comprises:

[0009] determining third audio data and fourth audio data in the target audio data, the third audio data being audio data within a first preset time before the first audio data, and the fourth audio data being audio data within a second preset time after the first audio data;

[0010] generating the second audio data according to the third audio data, the fourth audio data and fifth audio data in the first audio data except the invalid data.

[0011] In some embodiments of the present disclosure, the first preset time is the same as the second preset time; or,

[0012] the greater the first time difference between the start time of the invalid data and the start time of the first audio data, the smaller the first preset time; the greater the second time difference between the end time of the first audio data and the end time of the invalid data, the smaller the second preset time; or,

[0013] The first preset time and the second preset time are positively correlated with the duration of the invalid data.

[0014] In some embodiments of the present disclosure, the generating the second audio data according to at least part of the audio data in the target audio data excluding the invalid data comprises:

[0015] determining a start time and an end time of the invalid data;

[0016] determining sixth audio data in the target audio data according to the start time and the end time of the invalid data;

[0017] generating the second audio data according to the sixth audio data.

[0018] In some embodiments of the present disclosure, the determining the sixth audio data in the target audio data according to the start time and the end time of the invalid data comprises:

[0019] in a case where a first time difference between the start time of the invalid data and the start time of the first audio data and a second time difference between the end time of the first audio data and the end time of the invalid data are greater than or equal to a preset time difference, determining fifth audio data in the first audio data excluding the invalid data as the sixth audio data;

[0020] in a case where the first time difference is less than the preset time difference and the second time difference is greater than or equal to the preset time difference, determining audio data combined from third audio data and the fifth audio data as the sixth audio data, the third audio data being audio data within a first preset time before the first audio data;

[0021] in a case where the first time difference is greater than or equal to the preset time difference and the second time difference is less than the preset time difference, determining audio data combined from fourth audio data and the fifth audio data as the sixth audio data, the fourth audio data being audio data within a second preset time after the first audio data;

[0022] in a case where the first time difference and the second time difference are both less than the preset time difference, determining audio data combined from the third audio data, the fourth audio data and the fifth audio data as the sixth audio data.

[0023] In some embodiments of the present disclosure, the presence of the invalid data in the first audio data includes the presence of continuous preset data in the first audio data, and an absolute value of an amplitude corresponding to audio data before the continuous preset data is greater than or equal to a preset amplitude.

[0024] In some embodiments of the present disclosure, the invalid data is the continuous preset data.

[0025] In some embodiments of the present disclosure, a duration of the continuous preset data is greater than or equal to a third preset time.

[0026] In some embodiments of the present disclosure, the generating the second audio data according to at least part of the audio data in the target audio data except the invalid data includes:

[0027] inputting at least part of the audio data in the target audio data except the invalid data into a generative adversarial network model to obtain the second audio data.

[0028] In some embodiments of the present disclosure, the first audio data is audio data in an audio buffer.

[0029] According to a second aspect of embodiments of the present disclosure, an audio processing apparatus is provided, and the audio processing apparatus includes:

[0030] a determination module configured to determine first audio data to be played in target audio data;

[0031] a generation module configured to, in a case where invalid data is present in the first audio data, generate second audio data according to at least part of the audio data in the target audio data except the invalid data;

[0032] a replacement module configured to replace the invalid data in the first audio data with the second audio data.

[0033] According to a third aspect of embodiments of the present disclosure, an electronic device is provided, and the electronic device includes:

[0034] a processor;

[0035] a memory for storing processor-executable instructions;

[0036] wherein the processor is configured to perform the audio processing method as described above.

[0037] According to a fourth aspect of embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, and when instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to perform the audio processing method as described above.

[0038] The technical solutions provided by the embodiments of the present disclosure can include the following beneficial effects:

[0039] The first audio data to be played in the target audio data is determined to determine whether noise will occur when the first audio data is played. In the case that there is invalid data in the first audio data, noise will occur when the first audio data is played. The second audio data is generated according to at least part of the audio data in the target audio data except the invalid data, so as to obtain valid audio data. The invalid data in the first audio data is replaced by the second audio data, so that the second audio data is played instead of the invalid data when the invalid data is played. By generating the valid second audio data to replace the invalid data, the noise caused by playing the invalid data is avoided, so that the noise in the audio playing process is eliminated.

[0040] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure.

[0042] Figure 1 is a block diagram of an audio processing system according to an exemplary embodiment;

[0043] Figure 2 is a flowchart of an audio processing method according to an exemplary embodiment;

[0044] Figure 3 is a flowchart of an audio processing method according to another exemplary embodiment;

[0045] Figure 4 is a flowchart of an audio processing method according to another exemplary embodiment;

[0046] Figure 5 is a schematic diagram of target audio data according to an exemplary embodiment;

[0047] Figure 6 is a schematic diagram of target audio data according to another exemplary embodiment;

[0048] Figure 7 is a flowchart of an audio processing method according to another exemplary embodiment;

[0049] Figure 8 is a schematic diagram of target audio data according to another exemplary embodiment;

[0050] Figure 9 is a block diagram of an audio processing apparatus according to an example embodiment;

[0051] Figure 10 is a block diagram of an electronic device according to an example embodiment.

[0052] in the drawings:

[0053] 100 - determining module; 200 - generating module; 300 - replacing module; 400 - electronic device; 402 - processing component; 404 - memory; 406 - power component; 408 - multimedia component; 410 - audio component; 412 - input / output interface; 414 - sensor component; 416 - communication component; 420 - processor. DETAILED DESCRIPTION

[0054] The example embodiments will be described in detail herein with reference to the attached drawings. The following description is made with reference to the accompanying drawings in which like reference numerals refer to like elements, unless the context of use indicates otherwise. The following description of example embodiments is not representative of all embodiments consistent with the present disclosure. Rather, it is merely an example of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims. It should also be understood that the term "and / or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.

[0055] With the development of electronic devices, electronic devices can not only be used for communication, but also can be used for playing high-quality audio. For example, in the case where a music application, a video application, a game application, or the like is running, the electronic device can play audio generated by the application through a speaker, a headset, or the like to improve the user's experience. In the process of playing audio, due to the fact that the electronic device can simultaneously open multiple applications or the data processing capability of the electronic device is limited, the thread scheduling of the application writing audio data is not timely in a high-load scenario, so that there is certain invalid data in the audio. Or, due to the design defects of the application generating the audio, there is certain invalid data in the audio. Due to the fact that the audio suddenly jumps from valid audio data to invalid data in the process of playing, it causes pops and other noises, which affects the user's experience.

[0056] To solve the above technical problems, the present disclosure provides an audio processing method to generate valid second audio data from at least part of the target audio data except invalid data, and replace the invalid data in the first audio data to be played with the second audio data. When playing the first audio data, no invalid data jump occurs because there is no invalid data in the first audio data, thereby eliminating the noise in the audio playing process.

[0057] For ease of understanding, first, the audio processing system of the present disclosure is described. As shown in Figure 1 The audio processing system includes an application processor (AP) and a digital audio signal processor (ADSP). When the AP runs an application program, the thread scheduling of the application program writing audio data is not timely in a high load scenario, and the audio policy executor AudioFlinger in the audio component AudioServer for audio stream data processing and transmission cannot obtain valid audio data to fill in the audio data written to the audio hardware abstraction layer Audio HAL in the form of continuous preset data. The continuous preset data is invalid data. The preset data may be 0 data, for example. Or, due to design defects of the application program, the application program writes continuous preset data to the AudioFlinger through the framework AudioTrack for audio data output in the AudioServer in the process of generating audio. In the process of writing the audio data to the Audio HAL, there is invalid data in the audio data. After writing the audio data to the Audio HAL, the Audio HAL writes the audio data to the ADSP. After a series of processing, the audio data reaches the software mixer SW Mixer. The SW Mixer transmits the processed audio data to the task playback module Task Playback. The Task Playback processes the invalid data, converts the invalid data into valid audio data, and then plays through the speaker, earphone, etc. Since the Task Playback is the last module of the audio processing, it can maximize the guarantee that there is no invalid data in the audio data output to the playback device. The Task Playback can transmit the audio data to the playback device through the Inter-IC Sound (I2S) bus.

[0058] The present disclosure provides an audio processing method, as shown in Figure 2 The method comprises:

[0059] S100, determining the first audio data to be played in the target audio data.

[0060] S200, generating the second audio data according to at least part of the target audio data except the invalid data in the case that the invalid data exists in the first audio data.

[0061] S300, replacing the invalid data in the first audio data with the second audio data.

[0062] In the embodiment, the first audio data to be played in the target audio data is determined to determine whether the first audio data will produce noise when played. In the case that the invalid data exists in the first audio data, the first audio data will produce noise when played. The second audio data is generated according to at least part of the target audio data except the invalid data to obtain valid audio data. The invalid data in the first audio data is replaced with the second audio data so that the second audio data is played instead of the invalid data when the invalid data is played. By generating the valid second audio data to replace the invalid data, the noise caused by playing the invalid data is avoided, thereby eliminating the noise in the audio playing process.

[0063] Exemplarily, the target audio data and the second audio data can be generated by the audio processing system described above. The invalid data can exist in the target audio data or can not exist in the target audio data.

[0064] Exemplarily, the determination of the first audio data to be played in the target audio data in step S100 can be performed during the playing of the target audio data or before the playing of the target audio data.

[0065] In an embodiment, as shown in Figure 3 Exemplarily, the generation of the second audio data according to at least part of the target audio data except the invalid data in step S200 can be determined in the following manner:

[0066] S210, determining third audio data and fourth audio data in the target audio data, the third audio data being audio data within a first preset time before the first audio data, and the fourth audio data being audio data within a second preset time after the first audio data.

[0067] S220, generating the second audio data according to the third audio data, the fourth audio data and fifth audio data except the invalid data in the first audio data.

[0068] In the embodiment, the target audio data has certain correlation between the audio data before and after the target audio data to avoid the jump of the playing of the target audio data. Since there is audio data before and after the first audio data, the audio data within the first preset time before the first audio data and the audio data within the second preset time after the first audio data are determined. The second audio data is generated according to the third audio data, the fourth audio data and the fifth audio data, so that the second audio data is correlated with the audio data before and after the second audio data. The second audio data can play a transition role in the target audio data by generating the second audio data according to the audio data before and after the invalid data, thereby eliminating the noise in the audio playing process.

[0069] For example, the third audio data can be the audio data within the first preset time before the start time of the first audio data, and the fourth audio data can be the audio data within the second preset time after the end time of the first audio data.

[0070] In an embodiment, the first preset time and the second preset time are the same.

[0071] In the embodiment, the first preset time and the second preset time are set to be the same, so that the determination according to the start and end times of the invalid data and the first audio data is not required, thereby reducing the complexity of the audio processing.

[0072] For example, the first preset time and the second preset time can be in the range of 5ms to 50ms. The first preset time and the second preset time can be 10ms, 20ms, 30ms, etc.

[0073] In an embodiment, the greater the first time difference between the start time of the invalid data and the start time of the first audio data, the smaller the first preset time. The greater the second time difference between the end time of the first audio data and the end time of the invalid data, the smaller the second preset time.

[0074] In the embodiment, the greater the first time difference between the start time of the invalid data and the start time of the first audio data, the more the invalid data is biased to the second half of the first audio data without the third audio data with a long time, and the smaller the first preset time. The greater the second time difference between the end time of the first audio data and the end time of the invalid data, the more the invalid data is biased to the first half of the first audio data without the fourth audio data with a long time, and the smaller the second preset time. The first preset time and the second preset time are determined according to the position of the invalid data, so that the number of the audio data before and after the invalid data for generating the second audio data is relatively balanced, thereby improving the effect of the audio processing.

[0075] In an embodiment, the first preset time and the second preset time are positively correlated with the duration of the invalid data.

[0076] In the embodiment, by setting the first preset time and the second preset time to be positively correlated with the duration of the invalid data, the second audio data can be generated with less third audio data and fourth audio data in the case of less invalid data, thereby reducing the complexity of audio processing. Meanwhile, since the first preset time and the second preset time are positively correlated with the duration of the invalid data, the second audio data can be generated with more third audio data and fourth audio data in the case of more invalid data, thereby improving the effect of audio processing.

[0077] It can be understood that, in addition to generating the second audio data according to the third audio data, the fourth audio data and the fifth audio data which is the audio data other than the invalid data in the first audio data and the second audio data, the second audio data can also be generated according to only the third audio data and the fourth audio data.

[0078] In an embodiment, as shown in FIG. 2, the step S200 of generating the second audio data according to at least part of the audio data other than the invalid data in the target audio data can be determined in the following manner: Figure 4

[0079] S230, determining the start time and the end time of the invalid data.

[0080] S240, determining the sixth audio data in the target audio data according to the start time and the end time of the invalid data.

[0081] S250, generating the second audio data according to the sixth audio data.

[0082] In the embodiment, the start time and the end time of the invalid data are determined to determine the position of the invalid data and the duration of the invalid data. The sixth audio data in the target audio data is determined according to the start time and the end time of the invalid data, so as to determine the audio data for generating the second audio data based on the position and the duration of the invalid data. The second audio data is generated according to the sixth audio data, so that the second audio data can replace the corresponding invalid data. The second audio data is generated according to the sixth audio data determined based on the position and the duration of the invalid data, so as to reduce the audio data used for generating the second audio data, thereby reducing the complexity of audio processing.

[0083] In an embodiment, the step S240 of determining the sixth audio data in the target audio data according to the start time and the end time of the invalid data is determined in the following manner:

[0084] ​In a case where the first time difference between the start time of the invalid data and the start time of the first audio data and the second time difference between the end time of the first audio data and the end time of the invalid data are both greater than or equal to the preset time difference, fifth audio data in the first audio data except the invalid data is determined as the sixth audio data.

[0085] In a case where the first time difference is less than the preset time difference and the second time difference is greater than or equal to the preset time difference, audio data combined from the third audio data and the fifth audio data is determined as the sixth audio data, the third audio data being audio data within a first preset time before the first audio data.

[0086] In a case where the first time difference is greater than or equal to the preset time difference and the second time difference is less than the preset time difference, audio data combined from the fourth audio data and the fifth audio data is determined as the sixth audio data, the fourth audio data being audio data within a second preset time after the first audio data.

[0087] In a case where the first time difference and the second time difference are both less than the preset time difference, audio data combined from the third audio data, the fourth audio data and the fifth audio data is determined as the sixth audio data.

[0088] In the embodiment, in a case where the first time difference and the second time difference are both greater than or equal to the preset time difference, the invalid data is located in the middle section of the first audio data and has a short duration, the fifth audio data is determined as the sixth audio data. In a case where the first time difference is less than the preset time difference and the second time difference is greater than or equal to the preset time difference, the invalid data is biased to the first half of the first audio data and has a moderate duration, audio data combined from the third audio data and the fifth audio data is determined as the sixth audio data. In a case where the first time difference is greater than or equal to the preset time difference and the second time difference is less than the preset time difference, the invalid data is biased to the second half of the first audio data and has a moderate duration, audio data combined from the fourth audio data and the fifth audio data is determined as the sixth audio data. In a case where the first time difference and the second time difference are both less than the preset time difference, the invalid data is located in the middle section of the first audio data and has a long duration, audio data combined from the third audio data, the fourth audio data and the fifth audio data is determined as the sixth audio data. By determining the sixth audio data according to the first time difference and the second time difference, the audio data required for generating the second audio data is reduced in a case where the second audio data can be generated, thereby reducing the complexity of audio processing.

[0089] Exemplarily, the preset time difference can be determined according to the duration of the first audio data. The preset time difference can be in a range of 3 ms to 20 ms. The preset time difference can be 5 ms, 8 ms, 10 ms, etc.

[0090] In an embodiment, the presence of invalid data in the first audio data includes that there is continuous preset data in the first audio data, and an absolute value of an amplitude corresponding to audio data before the continuous preset data is greater than or equal to a preset amplitude. The invalid data is the continuous preset data.

[0091] In the embodiment, the audio data with noise suddenly jumps from a larger value to an amplitude corresponding to preset data in a playing process, and the invalid data lasts for a certain time. The presence of invalid data in the first audio data includes that there is continuous preset data in the first audio data, and an absolute value of an amplitude corresponding to audio data before the continuous preset data is greater than or equal to a preset amplitude. The presence of invalid data in the first audio data is determined according to the time duration of the preset data and the change amount of the amplitude, so that valid audio data is avoided from being determined as invalid data, and the reliability of audio processing is improved.

[0092] Exemplarily, the audio data before the continuous preset data can be one audio data before the continuous preset data, or a plurality of audio data before the continuous preset data.

[0093] Exemplarily, the preset amplitude can be in a range of 5 dB to 20 dB. The preset amplitude can be 8 dB, 10 dB, 12 dB, or the like.

[0094] In an embodiment, the continuous preset data lasts for a time greater than or equal to a third preset time.

[0095] In the embodiment, a small amount of preset data does not generate noise and there can also be a small amount of preset data in the first audio data without invalid data. The continuous preset data lasts for a time greater than or equal to a third preset time as a condition for the presence of invalid data in the first audio data to avoid incorrect replacement of audio data, and the reliability of audio processing is improved.

[0096] Exemplarily, the third preset time can be in a range of 2 ms to 10 ms. The third preset time can be 5 ms, 7 ms, 9 ms, or the like.

[0097] Exemplarily, as shown in FIG. 4, in a case where there is no invalid data in the first audio data, an amplitude corresponding to audio data gradually decreases to an amplitude corresponding to preset data in an oscillating manner. Figure 5 As shown in FIG. 5, in a case where there is invalid data in the first audio data, an amplitude corresponding to audio data suddenly jumps to an amplitude corresponding to preset data. In the figures, the horizontal axis represents time, and the vertical axis represents amplitude. The audio data in the window is the first audio data. Figure 6

[0098] ​In an embodiment, the generating the second audio data according to at least part of the audio data in the target audio data except the invalid data in the step S200 can be determined by the following way:

[0099] inputting the at least part of the audio data in the target audio data except the invalid data into the generative adversarial network model to obtain the second audio data.

[0100] In the embodiment, the second audio data is generated by the generative adversarial network model, which can generate real second audio data to replace the invalid data by combining the valid audio data, thereby improving the effect of audio processing. Meanwhile, the second audio data is generated by the generative adversarial network model without complex processing of the valid audio data, thereby reducing the complexity of audio processing.

[0101] For example, the inputting the at least part of the audio data in the target audio data except the invalid data into the generative adversarial network model to obtain the second audio data in the above steps can be inputting the third audio data, the fourth audio data and the fifth audio data into the generative adversarial network model to obtain the second audio data, or inputting the sixth audio data into the generative adversarial network model to obtain the second audio data.

[0102] In an embodiment, the first audio data is audio data in an audio buffer.

[0103] In the embodiment, the audio data in the audio buffer is audio data to be played, and the audio data in the audio buffer is processed to avoid noise in the audio data to be played, thereby improving the reliability of audio processing. Meanwhile, the first audio data is audio data outside the audio buffer, which avoids the processed audio data not being played due to the termination of the application, thereby further improving the reliability of audio processing.

[0104] The embodiments of the present disclosure provide an audio processing method, as shown in the following Figure 7 The method comprises the following steps:

[0105] S400, determining first audio data in an audio buffer.

[0106] S410, determining third audio data and fourth audio data in the target audio data in the case that there is continuous preset data in the first audio data and the absolute value of the amplitude corresponding to the audio data before the continuous preset data is greater than or equal to a preset amplitude.

[0107] S420, inputting the third audio data, the fourth audio data and fifth audio data except invalid data in the first audio data into a generative adversarial network model to obtain second audio data.

[0108] S430, replacing the invalid data in the first audio data with the second audio data.

[0109] S440, playing the second audio data in a case where a play time reaches a start time of the invalid data.

[0110] In this embodiment, the first audio data in the audio buffer is determined to determine whether the audio data to be played has invalid data. In a case where there is continuous preset data in the first audio data and the absolute value of the amplitude corresponding to the audio data before the continuous preset data is greater than or equal to a preset amplitude, it is determined that there is invalid data in the first audio data, and the third audio data and the fourth audio data located before and after the first audio data are determined. The third audio data, the fourth audio data and the fifth audio data are input into the generative adversarial network model, and the second audio data is obtained by generating through the generative adversarial network model. The invalid data in the first audio data is replaced with the second audio data, and the second audio data is played at the start time of the invalid data. By generating valid second audio data to replace invalid data, the noise caused by playing invalid data is avoided, thereby eliminating the noise in the audio playing process.

[0111] Exemplarily, as shown in Figure 6 and Figure 8 After the invalid data in the first audio data is replaced with the second audio data, the amplitude corresponding to the invalid data is in an oscillation state instead of jumping, thereby eliminating the noise in the audio playing process. Wherein, the horizontal axis represents time, and the vertical axis represents amplitude. The audio data in the window is the first audio data.

[0112] In one example embodiment, an audio processing apparatus is provided for implementing the above method. Referring to Figure 9 , the audio processing apparatus can include a determination module 100, a generation module 200 and a replacement module 300, wherein in the process of implementing the above method,

[0113] The determination module 100 is configured to determine the first audio data to be played in the target audio data.

[0114] The generation module 200 is configured to generate the second audio data according to at least part of the audio data in the target audio data except the invalid data in a case where there is invalid data in the first audio data.

[0115] The replacement module 300 is configured to replace the invalid data in the first audio data with the second audio data.

[0116] In one example embodiment, an audio processing apparatus is provided, wherein the generation module 200 is configured to:

[0117] determining third audio data and fourth audio data in the target audio data, the third audio data being audio data within a first preset time before the first audio data, and the fourth audio data being audio data within a second preset time after the first audio data.

[0118] generating the second audio data according to the third audio data, the fourth audio data, and fifth audio data in the first audio data excluding the invalid data.

[0119] In one example embodiment, there is provided an audio processing apparatus, in which a generating module 200 is configured to:

[0120] determining a start time and an end time of the invalid data.

[0121] determining sixth audio data in the target audio data according to the start time and the end time of the invalid data.

[0122] generating the second audio data according to the sixth audio data.

[0123] In one example embodiment, there is provided an audio processing apparatus, in which a generating module 200 is configured to:

[0124] in a case where a first time difference between the start time of the invalid data and a start time of the first audio data and a second time difference between an end time of the first audio data and an end time of the invalid data are both greater than or equal to a preset time difference, determining the fifth audio data in the first audio data excluding the invalid data as the sixth audio data.

[0125] in a case where the first time difference is less than the preset time difference and the second time difference is greater than or equal to the preset time difference, determining audio data combined from the third audio data and the fifth audio data as the sixth audio data, the third audio data being audio data within a first preset time before the first audio data.

[0126] in a case where the first time difference is greater than or equal to the preset time difference and the second time difference is less than the preset time difference, determining audio data combined from the fourth audio data and the fifth audio data as the sixth audio data, the fourth audio data being audio data within a second preset time after the first audio data.

[0127] in a case where the first time difference and the second time difference are both less than the preset time difference, determining audio data combined from the third audio data, the fourth audio data, and the fifth audio data as the sixth audio data.

[0128] In one example embodiment, there is provided an audio processing apparatus, in which a generating module 200 is configured to:

[0129] Input at least part of the audio data in the target audio data except the invalid data into the generative adversarial network model to obtain second audio data.

[0130] In one example embodiment, an electronic device, such as a mobile phone, a notebook computer, a tablet computer, a wearable device, and the like, is provided.

[0131] Reference Figure 10 As shown, the electronic device 400 can include one or more of the following components: a processing component 402, a memory 404, a power supply component 406, a multimedia component 408, an audio component 410, an input / output (I / O) interface 412, a sensor component 414, and a communication component 416.

[0132] The processing component 402 usually controls overall operations of the electronic device 400, such as operations associated with displaying, making phone calls, data communications, camera operations, and recording operations. The processing component 402 can include one or more processors 420 to execute instructions to complete all or part of steps of the methods described above. In addition, the processing component 402 can include one or more modules to facilitate the interaction between the processing component 402 and other components. For example, the processing component 402 can include a multimedia module to facilitate the interaction between the multimedia component 408 and the processing component 402.

[0133] The memory 404 is configured to store various types of data to support operations of the electronic device 400. Examples of these data include instructions for any application or method operating on the electronic device 400, contact data, phonebook data, messages, pictures, videos, and the like. The memory 404 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0134] The power supply component 406 provides power for various components of the electronic device 400. The power supply component 406 can include a power supply management system, one or more power supplies, and other components associated with generating, managing and distributing power for the electronic device 400.

[0135] The multimedia component 408 includes a screen to provide an output interface between the electronic device 400 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect duration and intensity of the touch or slide action. In some embodiments, the multimedia component 408 includes a front camera module and / or a rear camera module. The front camera module and / or the rear camera module can receive external multimedia data when the electronic device 400 is in an operation mode, such as a photographing mode or a video recording mode. Each of the front camera module and the rear camera module can be a fixed optical lens system or have a focal length and optical zoom ability.

[0136] The audio component 410 is configured to output and / or input an audio signal. For example, the audio component 410 includes a microphone (MIC) configured to receive an external audio signal when the electronic device 400 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 404 or transmitted via the communication component 416. In some embodiments, the audio component 410 further includes a speaker for outputting an audio signal.

[0137] The I / O interface 412 provides an interface between the processing component 402 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0138] The sensor component 414 includes one or more sensors to provide various state assessments for the electronic device 400. For example, the sensor component 414 can detect an open / closed state of the electronic device 400, relative positioning of components, such as a display and a keypad of the electronic device 400, a change in position of the electronic device 400 or a component of the electronic device 400, presence or absence of user contact with the electronic device 400, an orientation or acceleration / deceleration of the electronic device 400, and a temperature change of the electronic device 400. The sensor component 414 can include a proximity sensor configured to detect presence of a nearby object without any physical touch. The sensor component 414 can further include a light sensor, such as a CMOS or CCD image sensor, for use in an imaging application. In some embodiments, the sensor component 414 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0139] The communication component 416 is configured to facilitate wired or wireless communication between the electronic device 400 and other terminals. The electronic device 400 can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, or a combination thereof. In an exemplary embodiment, the communication component 416 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technology.

[0140] In an exemplary embodiment, the electronic device 400 can be implemented with one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing terminals (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements, for performing a method as shown in the above-described embodiments or a combination thereof.

[0141] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 404 including instructions, is also provided, which can be executed by the processor 420 of the electronic device 400 to implement a method as shown in the above-described embodiments or a combination thereof. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage terminal, etc. When the instructions in the storage medium are executed by the processor of the terminal, the terminal is enabled to perform a method as shown in the above-described embodiments or a combination thereof.

[0142] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure following, in general, the principles of the present disclosure and including such features to the extent that they are not disclosed in the prior art. The specification and examples given are intended as illustrative only and not restrictive on the true scope and spirit of the present disclosure, which is set forth in the following claims.

[0143] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, different embodiments or examples described in the specification and features of different embodiments or examples can be combined and combined by those skilled in the art without contradiction, if possible.

[0144] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0145] It should be understood that various forms of flow shown above can be reordered, added or deleted steps. For example, the steps described in the present disclosure can be executed in parallel, sequentially or in different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, which is not limited herein.

[0146] The above specific embodiments do not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. An audio processing method, characterized in that, The audio processing method includes: Determine the first audio data to be played from the target audio data; If invalid data exists in the first audio data, second audio data is generated based on at least a portion of the audio data in the target audio data excluding the invalid data. Replace the invalid data in the first audio data with the second audio data.

2. The audio processing method according to claim 1, characterized in that, The step of generating second audio data based on at least a portion of the audio data in the target audio data, excluding the invalid data, includes: The third audio data and the fourth audio data in the target audio data are determined. The third audio data is the audio data within a first preset time period before the first audio data, and the fourth audio data is the audio data within a second preset time period after the first audio data. The second audio data is generated based on the third audio data, the fourth audio data, and the fifth audio data in the first audio data excluding the invalid data.

3. The audio processing method according to claim 2, characterized in that, The first preset time is the same as the second preset time; or, The greater the first time difference between the start time of the invalid data and the start time of the first audio data, the smaller the first preset time; the greater the second time difference between the end time of the first audio data and the end time of the invalid data, the smaller the second preset time. or, Both the first preset time and the second preset time are positively correlated with the duration of the invalid data.

4. The audio processing method according to claim 1, characterized in that, The step of generating second audio data based on at least a portion of the audio data in the target audio data, excluding the invalid data, includes: Determine the start and end times of the invalid data; Based on the start and end times of the invalid data, determine the sixth audio data in the target audio data; The second audio data is generated based on the sixth audio data.

5. The audio processing method according to claim 4, characterized in that, The step of determining the sixth audio data in the target audio data based on the start and end times of the invalid data includes: If the first time difference between the start time of the invalid data and the start time of the first audio data and the second time difference between the end time of the first audio data and the end time of the invalid data are both greater than or equal to a preset time difference, the fifth audio data in the first audio data other than the invalid data is determined as the sixth audio data. When the first time difference is less than the preset time difference and the second time difference is greater than or equal to the preset time difference, the audio data obtained by combining the third audio data and the fifth audio data is determined as the sixth audio data, and the third audio data is the audio data within a first preset time period before the first audio data; When the first time difference is greater than or equal to the preset time difference and the second time difference is less than the preset time difference, the audio data obtained by combining the fourth audio data and the fifth audio data is determined as the sixth audio data, and the fourth audio data is the audio data within the second preset time period after the first audio data; If both the first time difference and the second time difference are less than the preset time difference, the audio data obtained by combining the third audio data, the fourth audio data, and the fifth audio data is determined as the sixth audio data.

6. The audio processing method according to claim 1, characterized in that, The presence of invalid data in the first audio data includes the presence of consecutive preset data in the first audio data, where the absolute value of the amplitude of the audio data preceding the consecutive preset data is greater than or equal to the preset amplitude. The invalid data refers to the continuous preset data.

7. The audio processing method according to claim 6, characterized in that, The duration of the continuous preset data is greater than or equal to a third preset time.

8. The audio processing method according to any one of claims 1 to 7, characterized in that, The step of generating second audio data based on at least a portion of the audio data in the target audio data, excluding the invalid data, includes: At least a portion of the audio data, excluding the invalid data, from the target audio data is input into a generative adversarial network model to obtain the second audio data.

9. The audio processing method according to any one of claims 1 to 7, characterized in that, The first audio data is the audio data in the audio buffer.

10. An audio processing apparatus, characterized in that, The audio processing device includes: The determining module is configured to determine the first audio data to be played in the target audio data; A generation module is configured to generate second audio data based on at least a portion of the audio data in the target audio data, excluding the invalid data, when invalid data exists in the first audio data; A replacement module, configured to replace the invalid data in the first audio data with the second audio data.

11. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to perform the audio processing method as described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the terminal, the terminal is able to perform the audio processing method as described in any one of claims 1 to 9.

Images