Audio processing method and device, electronic equipment and readable storage medium
By generating auxiliary decoding information and noise level information in audio processing, and iteratively processing the corrected audio data, the problem of low sound quality repair quality in the prior art is solved, and more stable audio processing effects and higher sound quality repair quality are achieved.
Patent Information
- Application Number
- CN202311070266.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-23
- Publication Date
- 2025-02-25
AI Technical Summary
The prior art is difficult to ensure the sound quality repair quality in audio processing, and the processing effect is unstable.
By generating auxiliary decoding information and noise level information, combining iterative sound quality data, the corrected audio data is processed, and the real content of the target audio is gradually approached.
The stability of audio processing and sound quality repair quality are improved, so that the audio data that has been repaired with sound quality gradually approaches the real content of the target audio.
Smart Images
Figure CN119517060A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to an audio processing method, device, electronic device and readable storage medium. Background Art
[0002] During the audio recording process, there are often problems with the recording environment or recording equipment, such as environmental noise, reverberation, and recording equipment limitations leading to inappropriate frequency response, which may cause varying degrees of noise or defects in the recorded audio, affecting the audio recording effect.
[0003] At present, for noise or defects in audio, sound quality restoration is usually performed based on deep learning algorithms. However, the processing effect of deep learning algorithms is not stable, and it is difficult to guarantee the sound quality restoration quality of audio data. Summary of the invention
[0004] In view of this, the embodiments of the present disclosure provide an audio processing method, device, electronic device and readable storage medium to solve the problem that the audio processing effect is poor and it is difficult to obtain high-quality audio.
[0005] In a first aspect, an embodiment of the present disclosure provides an audio processing method, comprising: obtaining a target audio to be processed, and generating auxiliary decoding information based on the target audio; for a current iteration round in the audio processing process, obtaining noise level information and audio data to be corrected corresponding to the current iteration round; using the auxiliary decoding information and the noise level information as reference information, processing the audio data to be corrected, so as to generate iterative sound quality data corresponding to the audio data to be corrected; wherein the iterative sound quality data is used to generate an audio processing result of the target audio, or to generate audio data to be corrected for the next iteration round.
[0006] The audio processing method provided by the embodiment of the present disclosure generates auxiliary decoding information of the target audio to be processed, combines the auxiliary decoding information and the noise level information in the audio processing process, processes the audio data to be corrected, and generates iterative sound quality data. As a result, the target audio to be processed can be repaired for sound quality under the joint action of the auxiliary decoding information and the noise level as the iteration rounds continue, so that the audio data after sound quality repair gradually approaches the real content of the target audio. Under the action of the auxiliary decoding information, it can guide and supervise the processing process of the target audio, so as to obtain high-quality audio with the same content as the target audio representation, making the audio processing process more stable, further improving the sound quality of the audio, and ensuring the quality of sound quality repair.
[0007] In a second aspect, an embodiment of the present disclosure provides an audio processing device, which includes: a target audio processing unit, used to obtain a target audio to be processed and generate auxiliary decoding information based on the target audio; a round data acquisition unit, used to obtain, for a current iteration round in an audio processing process, noise level information and audio data to be corrected corresponding to the current iteration round; an iterative sound quality data generation unit, used to process the audio data to be corrected using the auxiliary decoding information and the noise level information as reference information, so as to generate iterative sound quality data corresponding to the audio data to be corrected; wherein the iterative sound quality data is used to generate an audio processing result of the target audio, or to generate audio data to be corrected for the next iteration round.
[0008] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a memory and a processor, the memory being used to store a computer program, and when the computer program is executed by the processor, an audio processing method as in the first aspect or any one of the embodiments of the first aspect is implemented.
[0009] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it implements the audio processing method as described in the first aspect or any implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0011] Figure 1 is a flowchart of an audio processing method according to an embodiment of the present disclosure;
[0012] Figure 2 is a schematic diagram of an audio restoration model according to an embodiment of the present disclosure;
[0013] Figure 3 is a flowchart of another audio processing method according to an embodiment of the present disclosure;
[0014] Figure 4 is a structural block diagram of an audio processing device according to an embodiment of the present disclosure;
[0015] Figure 5 It is a schematic diagram of the hardware structure of the electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0016] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.
[0017] In the related technologies, deep learning algorithms for pre-processing audio data can be divided into two categories: the first category is the sound quality improvement algorithm based on regression network, but this type of solution is often used for tasks such as noise reduction / dereverberation, and is prone to introduce defects in spectrum repair and super-resolution; the second category is the sound quality improvement algorithm based on generative network, but this type of solution has a high degree of instability in processing effects for more complex input audio, such as excessive reverberation noise, multiple speakers, etc. Therefore, the current audio data beautification is only a single-point processing, and it is difficult to perform good sound quality repair on audio data.
[0018] Based on this, the technical solution disclosed in the present invention utilizes the regression network sound quality improvement model in the frequency domain as a conditional network, and provides additional information to the generative network part for auxiliary generation, so as to have multiple repair capabilities, thereby better repairing the audio data.
[0019] According to an embodiment of the present disclosure, an embodiment of an audio processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0020] In this embodiment, an audio processing method is provided, which can be used in electronic devices such as computers, servers, etc. Figure 1 is a flowchart of an audio processing method according to an embodiment of the present disclosure, such as Figure 1 As shown, the process includes the following steps:
[0021] Step S101, obtaining target audio to be processed, and generating auxiliary decoding information based on the target audio.
[0022] The target audio is audio with poor sound quality. The target audio may be pre-recorded audio with various noise interferences, or may be audio randomly selected from a pre-built audio sample library. The method for obtaining the target audio is not specifically limited here.
[0023] The auxiliary decoding information is a parameter output after encoding and decoding the target audio, and the auxiliary decoding information indicates the restoration trend of the target audio during the restoration process. Specifically, the auxiliary decoding information is a parameter output by the pre-trained second branch model during the restoration process of the target audio, such as Figure 2 The second branch model can be a regression network model, which takes the target audio Y as input and uses high-quality audio with the same content as the target audio As output, the trained, auxiliary decoding information is output by the decoding convolutional layer.
[0024] Step S102: for a current iteration round in the audio processing process, obtain noise level information and audio data to be corrected corresponding to the current iteration round.
[0025] In order to obtain high-quality audio data, multiple rounds of iterative processing are required. For each iteration round, there are corresponding noise level information and audio data to be corrected. Among them, the noise level information is the noise of different degrees superimposed within a preset range; the audio data to be corrected is the audio data output after audio repair processing under the current noise level information. In the first round of iteration, the audio data to be corrected is a Gaussian white noise. As the iteration rounds continue, under the supervision of various signals, the Gaussian white noise can be processed into the required high-quality audio data.
[0026] Specifically, Figure 2 As shown, the audio processing process can be implemented by a pre-trained first branch model, which can be a generative network model. The first branch model and the second branch model constitute an audio restoration model.
[0027] The input of the first branch model is low-quality audio data obtained by superimposing different degrees of noise within a preset range on the spectrum of high-quality audio. At the same time, different degrees of noise are used as conditional inputs of the first branch model to estimate the score S. t , we can use geometric series according to the preset σ min , σ max To generate different levels of noise, as follows:
[0028] σ t =σ min (σ max / σ min ) t
[0029] Here, a stochastic differential equation can be used to make t a continuous value during the training process. Finally, the first branch model is repeatedly corrected and trained through the loss function, so that the first branch model can finally output the estimated score S in the frequency domain. The expression of the loss function is:
[0030]
[0031] Among them, x0 is high-quality audio, z t and σ t All are noise, x0+σ t z t is low-quality audio with noise superimposed on it, c is auxiliary decoding information, and E represents expectation.
[0032] Step S103: Using the auxiliary decoding information and the noise level information as reference information, the audio data to be corrected is processed to generate iterative sound quality data corresponding to the audio data to be corrected.
[0033] The iterative sound quality data is used to generate an audio processing result of the target audio, or to generate audio data to be corrected for the next iterative round.
[0034] Specifically, the audio data to be corrected is iteratively processed in combination with the consistent annealing algorithm to obtain the corresponding iterative sound quality data. Taking the first iteration as an example, the input is Gaussian white noise, which generates an estimated score S under the joint action of the noise level information and the auxiliary decoding information. Then, the audio data to be corrected for the next iteration is generated by combining the Gaussian white noise and the estimated score S. Through repeated iterations, high-quality audio data with the same content as the target audio representation can be obtained.
[0035] In this process, the auxiliary decoding information can guide and supervise the processing method of the audio data to be corrected (Gaussian white noise in the first round), so that the estimated score S can gradually approach the true content of the target audio, that is, realize the conversion from noise to high-quality audio.
[0036] The audio processing method provided in this embodiment generates auxiliary decoding information of the target audio to be processed, combines the auxiliary decoding information and the noise level information in the audio processing process, processes the audio data to be corrected, and generates iterative sound quality data. As a result, the target audio to be processed can be repaired for sound quality under the joint action of the auxiliary decoding information and the noise level as the iteration rounds continue, so that the audio data after sound quality repair gradually approaches the real content of the target audio. Under the action of the auxiliary decoding information, it can guide and supervise the processing process of the target audio, so as to obtain high-quality audio with the same content as the target audio representation, making the audio processing process more stable, further improving the sound quality of the audio, and ensuring the quality of sound quality repair.
[0037] In this embodiment, an audio processing method is provided, which can be used in electronic devices such as computers, servers, etc. Figure 3 is a flowchart of an audio processing method according to an embodiment of the present disclosure, such as Figure 3 As shown, the process includes the following steps:
[0038] Step S201: Acquire target audio to be processed, and generate auxiliary decoding information based on the target audio.
[0039] Specifically, the above step S201 may include:
[0040] Step S2011, obtain the target audio to be processed, use one or more encoding convolution layers to process the target audio, and generate encoded data of the target audio.
[0041] like Figure 2 As shown, the second branch model includes an encoding unit and a decoding unit, and the encoding unit and the decoding unit are linked by a hidden convolution unit. The encoding unit has one or more encoding convolution layers; the hidden convolution unit can map the encoded data to a higher-dimensional feature space, and help the second branch model better understand the audio data by learning and extracting the features of the encoded data. The hidden convolution unit has one or more hidden convolution layers, and each hidden convolution layer can learn abstract features at different levels.
[0042] After acquiring the target audio, it is input into the encoding unit of the second branch model to perform encoding convolution processing on the target audio through one or more encoding convolution layers to extract features in the target audio and generate corresponding encoding data.
[0043] Specifically, when processing the target audio using one or more encoding convolutional layers, the above method may also include: obtaining the encoded intermediate data output by each encoding convolutional layer, and converting each encoded intermediate data to the same data dimension.
[0044] In order to ensure that the encoded intermediate data of different sampling dimensions are flattened to the same data dimension, so as to facilitate the subsequent superposition of the encoded intermediate data, the activation layer PReLU and the dimension conversion layer StConv can be set for each encoding convolution layer, such as Figure 2 After each encoding convolution layer completes the encoding process, the encoded intermediate data outputted by it can be nonlinearly transformed through the activation layer PReLU, and the dimension conversion layer StConv is used to convert the nonlinear transformation data outputted by the activation layer, so that all the encoded intermediate data are in the same data dimension.
[0045] Accordingly, after generating the encoded data of the target audio, the above method further includes: fusing the encoded intermediate data of the same data dimension with the encoded data, so as to process the fused data through one or more decoding convolutional layers.
[0046] The encoded data output by multiple encoding convolutional layers is fused with the encoded intermediate data output by each encoding convolutional layer so that the fused data can contain both shallow and deep features, thereby improving the accuracy of feature extraction. Furthermore, the fused data is mapped to the decoding convolutional unit through the hidden convolution unit for decoding processing to extract the audio data features for audio recovery.
[0047] In this embodiment, each encoded intermediate data is converted to the same data dimension so that the fused data can contain more features, thereby ensuring the accuracy of data processing.
[0048] Step S2012: Process the encoded data using one or more decoding convolutional layers, and use the decoded intermediate data output by each decoding convolutional layer as auxiliary decoding information of the target audio.
[0049] like Figure 2 As shown, the decoding unit has one or more decoding convolution layers. The number of decoding convolution layers is the same as that of encoding convolution layers. The encoded data obtained by the encoding unit is input into the decoding convolution layer in the decoding unit for decoding processing to obtain data features for the target audio, and then the audio is restored according to the data features.
[0050] At the same time, during the decoding convolution process of each decoding convolution layer, the corresponding decoding intermediate data will be output. The decoding intermediate data can characterize the repair status of the corresponding decoding convolution layer for the target audio. The decoding intermediate data is used as auxiliary decoding information to guide and supervise the first branch model for the repair of low-quality audio data.
[0051] Step S202: for the current iteration round in the audio processing process, obtain the noise level information and the audio data to be corrected corresponding to the current iteration round.
[0052] Specifically, the above step S202 may include:
[0053] Step S2021, obtaining a preset maximum noise level and a minimum noise level, and based on the total iteration rounds in the audio processing process, sampling the noise level interval composed of the maximum noise level and the minimum noise level to generate noise level information corresponding to each iteration round.
[0054] The total number of iterations is the total number of iterations required in the audio processing process, such as 6 times, 8 times, etc. Specifically, the total number of iterations is determined according to the training process of the first branch model. If, during the training process, 6 iterations are taken from low-quality audio to high-quality audio, then in the subsequent audio processing process, the total number of iterations is 6 times.
[0055] The maximum noise level and the minimum noise level are pre-set audio noise information, and different levels of noise are represented by the noise level interval formed by the minimum noise level and the maximum noise level. After determining the total number of iterations in the audio processing process, uniform sampling is performed within the noise level range represented by the minimum noise level and the maximum noise level according to the total number of iterations to obtain the noise level information corresponding to each iteration.
[0056] For example, the maximum noise level is A, the minimum noise level is B, and the total number of iterations is 6. Then 6 samples can be taken in the noise level interval [B, A] to obtain 6 sampling values. Generally speaking, B and A will be sampled, so that during the iteration process, the noise level information gradually decreases from A to B.
[0057] By sampling the intervals determined by the maximum noise level and the minimum noise level through the total iteration rounds, the noise level information corresponding to each iteration round is determined, thereby determining different degrees of noise, which facilitates the subsequent effective removal of noise in the target audio to obtain high-quality audio.
[0058] Step S2022: If the current iteration round is the first iteration round, the preset noise data is used as the audio data to be corrected corresponding to the current iteration round.
[0059] The preset noise data is a preset low-quality audio, that is, the audio data to be processed that is first input to the first branch model. The low-quality audio is superimposed with a variety of noises, which can be based on high-quality audio data, and the high-quality audio data is augmented with different data, including noise addition, reverberation, equalization, volume mutation, etc., to obtain simulated low-quality audio.
[0060] There are multiple iterations in the audio processing process, and the audio data to be corrected input in each iteration is different. Therefore, each time a new iteration is entered, it is necessary to detect whether the current iteration is the first iteration. If the current iteration is the first iteration, the data to be corrected input to the network is the preset noise data.
[0061] Step S2023, if the current iteration round is not the first iteration round, obtain the historical audio data to be corrected corresponding to the previous iteration round before the current iteration round, and the historical iteration sound quality data processed by the previous iteration round, and generate the audio data to be corrected corresponding to the current iteration round based on the historical audio data to be corrected and the historical iteration sound quality data.
[0062] If the current iteration round is not the first iteration round, it means that the audio data to be corrected in the current iteration round is determined based on the historical audio data to be corrected and the historical iteration sound quality data outputted in the previous iteration round. At this time, the historical audio data to be corrected and the historical iteration sound quality data corresponding to the previous iteration round before the current iteration round can be obtained, and the historical audio data to be corrected and the historical iteration sound quality data can be used as the current input, and the current audio data is processed to output the audio data to be corrected corresponding to the current iteration round.
[0063] By detecting whether the current iteration round is the first iteration round to determine the corresponding audio data to be corrected, the processing accuracy of the audio data to be corrected is improved.
[0064] Step S203: Using the auxiliary decoding information and the noise level information as reference information, the audio data to be corrected is processed to generate iterative sound quality data corresponding to the audio data to be corrected.
[0065] The iterative sound quality data is used to generate an audio processing result of the target audio, or to generate audio data to be corrected for the next iterative round.
[0066] Specifically, in the process of processing the audio data to be corrected, one or more encoding convolution layers and one or more decoding convolution layers are used, wherein the decoding convolution layers used in the process of processing the audio data to be corrected correspond one to one with the decoding convolution layers used in the process of generating the auxiliary decoding information, such as Figure 2 shown.
[0067] Accordingly, the above step S203 may include:
[0068] Step S2031, inputting the noise level information into each encoding convolution layer and each decoding convolution layer in the process of processing the audio data to be corrected.
[0069] like Figure 2As shown, the first branch model on the left includes corresponding encoding units and decoding units. In each iteration round, the corresponding noise level information is input into each encoding convolution layer of the encoding unit and each decoding convolution layer of the decoding unit through a linear layer.
[0070] In order to reduce the noise level information from the high dynamic range to the low dynamic range to improve stability, the noise level information can be logarithmically compressed. For example, the noise σ t Compress to log 10 (σ t ).
[0071] Step S2032, inputting the decoded intermediate data output by the decoding convolution layer in the auxiliary decoding information generation process into the corresponding decoding convolution layer in the audio data processing process to be corrected.
[0072] like Figure 2 As shown, the second branch model on the right will output the decoding intermediate data in the decoding convolution layer during the process of processing the target audio. As mentioned above, the decoding intermediate data is the auxiliary decoding information. The auxiliary decoding information can guide and supervise the sound quality restoration method of the first branch model for the audio data to be corrected. Therefore, the auxiliary decoding information generated by the second branch model on the right is input into the decoding convolution layer of the first branch model on the left to assist the processing of the audio data to be corrected.
[0073] In this embodiment, since the decoding convolution layer used in the processing of the audio data to be corrected corresponds one-to-one to the decoding convolution layer used in the auxiliary decoding information generation process, the symmetrical decoding convolution layer can be used to input the decoded intermediate data into the corresponding decoding convolution layer in the processing of the audio data to be corrected, and then the decoded intermediate data can be used as auxiliary information of the noise level information to process the audio data to be corrected, thereby improving the correction effect of the audio data.
[0074] In some optional implementations, the step of generating iterative sound quality data corresponding to the audio data to be corrected may include:
[0075] Step a1, when the noise level information is used as reference information, one or more coding convolutional layers are used to process the audio data to be corrected to generate coded data of the audio data to be corrected.
[0076] Step a2, when the auxiliary decoding information and the noise level information are used as reference information, the encoded data is processed using one or more decoding convolutional layers to generate iterative sound quality data corresponding to the audio data to be corrected.
[0077] like Figure 2As shown, the first branch model includes an encoding unit and a decoding unit, and the encoding unit and the decoding unit are linked by a hidden convolution unit. The encoding unit has one or more encoding convolution layers; the decoding unit has one or more decoding convolution layers, and the number of layers of the encoding convolution layer and the decoding convolution layer is the same; the hidden convolution unit can map the encoded data corresponding to the audio data to be corrected to a higher-dimensional feature space, and help the first branch model better understand the audio data to be corrected by learning and extracting the features of the encoded data. The hidden convolution unit has one or more hidden convolution layers, and each hidden convolution layer can learn abstract features at different levels.
[0078] After obtaining the audio data to be corrected, the audio data to be corrected and the noise level information are input into the encoding unit of the first branch model together, so that the audio data to be corrected is convolved through one or more encoding convolution layers under the action of the noise level information to extract the audio features in the audio data to be corrected and generate the encoding data corresponding to the audio data to be corrected.
[0079] Then, the above-obtained encoded data corresponding to the audio data to be corrected and the auxiliary decoding information generated by the regression repair model are input into the decoding unit of the first branch model, and decoding processing is performed through the decoding convolution layer in the decoding unit to obtain data features for the audio data to be corrected, and then an estimated score S in the frequency domain is output according to the data features. The estimated score and the consistent annealing algorithm are combined to generate corresponding iterative sound quality data.
[0080] Through the combined effect of auxiliary decoding information and noise level information, the audio data to be corrected is iteratively processed so that the audio data to be corrected can be corrected quickly, thereby improving the iterative efficiency of the audio data to be corrected.
[0081] In some optional implementations, the auxiliary decoding information and the iterative sound quality data are both generated by a trained joint model, where the joint model includes a first branch model and a second branch model.
[0082] Among them, the second branch model is used to generate auxiliary decoding information of the target audio and provide the auxiliary decoding information to the first branch model; the first branch model is used to execute each iterative round in the audio processing process while using the auxiliary decoding information and noise level information as reference information.
[0083] The joint model includes a first branch model and a second branch model, wherein the first branch model can be a generative network model and the second branch model can be a regression network model. The first branch model and the second branch model are jointly trained to obtain a corresponding joint model, which is Figure 2 The audio repair model shown.
[0084] Combination Figure 2 It can be seen that the second branch model is mainly used to provide auxiliary decoding information, which is generated by the second branch model in the process of repairing the target audio. The first branch model can determine the repair method or repair trend for the audio data to be corrected based on the auxiliary decoding information provided by the second branch model, thereby combining the noise level information and the auxiliary decoding information to determine the iteration information of each iteration round in the audio processing process.
[0085] Here, the iteration effect is improved by adopting a joint model to determine the auxiliary decoding information and the iterative sound quality data.
[0086] In some optional implementations, after generating iterative sound quality data corresponding to the audio data to be corrected, the method may further include:
[0087] Step b1: If the current iteration round is the last iteration round, generate an audio processing result of the target audio based on the audio data to be corrected and the iterative sound quality data.
[0088] Step b2: if the current iteration round is not the last iteration round, based on the audio data to be corrected and the iterative sound quality data, generate the audio data to be corrected for the next iteration round.
[0089] There are multiple iterations in the audio processing process, and the processed audio data output by each iteration is different. The audio data output by the last iteration is used to generate the final audio processing result. Therefore, each time a new iteration is entered, it is necessary to detect whether the current iteration is the last iteration. If the current iteration is the last iteration, the audio processing result is generated based on the audio data to be corrected and the iterative sound quality data output by the current iteration. If the current iteration is not the last iteration, it means that it is necessary to continue to the next iteration. At this time, the audio data to be corrected and the iterative sound quality data output by the current iteration are used as the input of the next iteration to generate the audio data to be corrected for the next iteration.
[0090] Here, the final audio processing effect is determined by detecting whether the current iteration round is the last iteration round, thereby ensuring that the audio data to be corrected can be restored to sound quality through a complete iteration process, thereby ensuring the quality of sound restoration.
[0091] The audio processing method provided in this embodiment combines the encoding convolution layer and the decoding convolution layer to encode and decode the target audio to generate auxiliary decoding information, which is convenient for guiding and supervising the iterative process of the target audio through the auxiliary decoding information, so that the iterative process of the target audio can converge quickly, thereby improving the processing efficiency of the target audio.
[0092] In this embodiment, an audio processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0093] This embodiment provides an audio processing device, such as Figure 4 As shown, the device comprises:
[0094] The target audio processing unit 301 is used to obtain the target audio to be processed and generate auxiliary decoding information based on the target audio.
[0095] The round data acquisition unit 302 is used to acquire the noise level information and the audio data to be corrected corresponding to the current iteration round in the audio processing process.
[0096] The iterative sound quality data generating unit 303 is used to process the audio data to be corrected using the auxiliary decoding information and the noise level information as reference information to generate iterative sound quality data corresponding to the audio data to be corrected. The iterative sound quality data is used to generate an audio processing result of the target audio, or to generate audio data to be corrected for the next iteration.
[0097] In some optional implementations, the target audio processing unit 301 may include:
[0098] The coding convolution module is used to process the target audio using one or more coding convolution layers to generate coded data of the target audio.
[0099] The decoding convolution module is used to process the encoded data using one or more decoding convolution layers, and use the decoded intermediate data output by each decoding convolution layer as auxiliary decoding information of the target audio.
[0100] In some optional implementations, the round data acquisition unit 302 may include:
[0101] The noise acquisition module is used to obtain the preset maximum noise level and minimum noise level, and based on the total iteration rounds in the audio processing process, sample the noise level interval composed of the maximum noise level and the minimum noise level to generate the noise level information corresponding to each iteration round.
[0102] The first determination module is used to use the preset noise data as the audio data to be corrected corresponding to the current iteration round if the current iteration round is the first iteration round.
[0103] The second judgment module is used to obtain the historical audio data to be corrected corresponding to the previous iteration round before the current iteration round, and the historical iteration sound quality data processed by the previous iteration round if the current iteration round is not the first iteration round, and generate the audio data to be corrected corresponding to the current iteration round based on the historical audio data to be corrected and the historical iteration sound quality data.
[0104] In some optional implementations, in the process of processing the audio data to be corrected, one or more encoding convolution layers and one or more decoding convolution layers are used, wherein the decoding convolution layers used in the process of processing the audio data to be corrected correspond one to one with the decoding convolution layers used in the process of generating the auxiliary decoding information. Accordingly, the above-mentioned iterative sound quality data generating unit 303 may include:
[0105] The first information input module is used to input the noise level information into each encoding convolution layer and each decoding convolution layer in the process of processing the audio data to be corrected.
[0106] The second information input module is used to input the decoded intermediate data output by the decoding convolution layer in the auxiliary decoding information generation process into the corresponding decoding convolution layer in the audio data processing process to be corrected.
[0107] In some optional implementations, the iterative sound quality data generating unit 303 may further include:
[0108] The coding generation module is used to process the audio data to be corrected by using one or more coding convolution layers with the noise level information as reference information to generate coding data of the audio data to be corrected.
[0109] The iterative data generation module is used to process the encoded data using one or more decoding convolutional layers when the auxiliary decoding information and the noise level information are used as reference information to generate iterative sound quality data corresponding to the audio data to be corrected.
[0110] In some optional implementations, the audio processing device further includes:
[0111] The joint training module is used to train the joint model according to the first branch model and the second branch model. The auxiliary decoding information and iterative sound quality data generated by the joint model are generated by the trained joint model.
[0112] Among them, the second branch model is used to generate auxiliary decoding information of the target audio and provide the auxiliary decoding information to the first branch model; the first branch model is used to execute each iterative round in the audio processing process while using the auxiliary decoding information and noise level information as reference information.
[0113] In some optional implementations, the audio processing device may further include:
[0114] The third determination module is used to generate an audio processing result of the target audio based on the audio data to be corrected and the iterative sound quality data if the current iteration round is the last iteration round.
[0115] The fourth determination module is used to generate the audio data to be corrected for the next iteration round based on the audio data to be corrected and the iterative sound quality data if the current iteration round is not the last iteration round.
[0116] The specific processing logic of each module and each unit can be found in the description of the aforementioned method implementation method, which will not be repeated here.
[0117] Each unit described in the above embodiments may be implemented by a computer chip or a product having a certain function. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0118] For the convenience of description, the above devices are described in terms of functions and are described separately in various units. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0119] In the audio processing device provided by this embodiment, the target audio to be processed can perform sound quality restoration on the target audio under the joint action of the auxiliary decoding information and the noise level as the iteration rounds continue, so that the audio data after sound quality restoration gradually approaches the real content of the target audio. Under the action of the auxiliary decoding information, it can guide and supervise the processing process of the target audio, making the audio processing process more stable, further improving the sound quality of the audio, and ensuring the quality of sound quality restoration.
[0120] The present disclosure also provides an electronic device having Figure 4 The audio processing device shown.
[0121] See also Figure 5 , Figure 5 is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present disclosure, such as Figure 5As shown, the electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.
[0122] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0123] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.
[0124] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0125] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0126] The electronic device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.
[0127] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.
[0128] The electronic device also includes a communication interface for data communication between the electronic device and other devices or a communication network.
[0129] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0130] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, equipment, and storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0131] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.
[0132] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. An audio processing method, characterized in that: The method comprises: Acquire target audio to be processed, and generate auxiliary decoding information based on the target audio; For a current iteration round in an audio processing process, obtaining noise level information and audio data to be corrected corresponding to the current iteration round; Using the auxiliary decoding information and the noise level information as reference information, the audio data to be corrected is processed to generate iterative sound quality data corresponding to the audio data to be corrected; The iterative sound quality data is used to generate the audio processing result of the target audio, or to generate the audio data to be corrected for the next iterative round.
2. The method according to claim 1, characterized in that: Generating auxiliary decoding information based on the target audio includes: Processing the target audio using one or more coding convolutional layers to generate coded data of the target audio; The encoded data is processed using one or more decoding convolutional layers, and the decoded intermediate data output by each of the decoding convolutional layers is used as auxiliary decoding information of the target audio.
3. The method according to claim 2, characterized in that When the target audio is processed using one or more encoding convolutional layers, the method further includes: Obtaining the encoded intermediate data output by each of the encoded convolutional layers, and converting each of the encoded intermediate data to the same data dimension; Accordingly, after generating the encoded data of the target audio, the method further includes: The encoded intermediate data of the same data dimension are fused with the encoded data, so as to process the fused data through the one or more decoding convolutional layers.
4. The method according to claim 2, characterized in that: In the process of processing the audio data to be corrected, one or more encoding convolution layers and one or more decoding convolution layers are used, wherein the decoding convolution layers used in the process of processing the audio data to be corrected correspond one-to-one to the decoding convolution layers used in the process of generating the auxiliary decoding information; Using the auxiliary decoding information and the noise level information as reference information, processing the audio data to be corrected includes: Inputting the noise level information into each encoding convolution layer and each decoding convolution layer in the process of processing the audio data to be corrected; and The decoded intermediate data output by the decoding convolution layer in the auxiliary decoding information generation process is input into the corresponding decoding convolution layer in the audio data processing process to be corrected.
5. The method according to claim 1, characterized in that The noise level information corresponding to the current iteration round is determined in the following manner: A preset maximum noise level and a minimum noise level are obtained, and based on the total iteration rounds in the audio processing process, a noise level interval consisting of the maximum noise level and the minimum noise level is sampled to generate noise level information corresponding to each iteration round.
6. The method according to claim 1, characterized in that The audio data to be corrected corresponding to the current iteration round is generated in the following manner: If the current iteration round is the first iteration round, using the preset noise data as the audio data to be corrected corresponding to the current iteration round; If the current iteration round is not the first iteration round, obtain the historical audio data to be corrected corresponding to the previous iteration round before the current iteration round, and the historical iteration sound quality data obtained by processing the previous iteration round, and generate the audio data to be corrected corresponding to the current iteration round based on the historical audio data to be corrected and the historical iteration sound quality data.
7. The method according to claim 1, characterized in that Generating iterative sound quality data corresponding to the audio data to be corrected includes: In a case where the noise level information is used as reference information, the audio data to be corrected is processed using one or more coding convolutional layers to generate coded data of the audio data to be corrected; In a case where the auxiliary decoding information and the noise level information are used together as reference information, the encoded data is processed using one or more decoding convolutional layers to generate iterative sound quality data corresponding to the audio data to be corrected.
8. The method according to claim 1, characterized in that The auxiliary decoding information and the iterative sound quality data are both generated by a trained joint model, wherein the joint model includes a first branch model and a second branch model; The second branch model is used to generate auxiliary decoding information of the target audio, and provide the auxiliary decoding information to the first branch model; The first branch model is used to perform various iteration rounds in the audio processing process with the auxiliary decoding information and the noise level information as reference information.
9. The method according to claim 1, characterized in that: After generating iterative sound quality data corresponding to the audio data to be corrected, the method further includes: If the current iteration round is the last iteration round, generating an audio processing result of the target audio based on the audio data to be corrected and the iterative sound quality data; If the current iteration round is not the last iteration round, the audio data to be corrected for the next iteration round is generated based on the audio data to be corrected and the iterative sound quality data.
10. An audio processing device, characterized in that: The device comprises: A target audio processing unit, used to obtain target audio to be processed and generate auxiliary decoding information based on the target audio; A round data acquisition unit, used for acquiring noise level information and audio data to be corrected corresponding to the current iteration round in the audio processing process; An iterative sound quality data generating unit is used to process the audio data to be corrected using the auxiliary decoding information and the noise level information as reference information, so as to generate iterative sound quality data corresponding to the audio data to be corrected; wherein the iterative sound quality data is used to generate an audio processing result of the target audio, or to generate the audio data to be corrected for the next iteration round.
11. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory is used to store a computer program, and when the computer program is executed by the processor, the audio processing method according to any one of claims 1 to 9 is implemented.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the audio processing method according to any one of claims 1 to 9 is implemented.