Audio processing method, device, electronic device and storage medium

Abstract

The embodiment of the present application discloses an audio processing method, device, electronic device and storage medium. The method includes: obtaining audio to be processed; dividing the audio to be processed into frames to obtain multiple audio frames; performing frequency domain transformation on each audio frame to obtain a first spectrogram and a second spectrogram of each audio frame; obtaining a first reference parameter and a second reference parameter according to the encoding parameters of the audio to be processed; obtaining a third spectrogram of each audio frame according to the first spectrogram and the first reference parameter of each audio frame; obtaining a fourth spectrogram of each audio frame according to the first spectrogram, the second spectrogram and the second reference parameter of each audio frame; fusing the third spectrogram and the fourth spectrogram of each audio frame to obtain a target spectrogram of each audio frame; performing time domain transformation on multiple target spectrograms corresponding to multiple audio frames to obtain target audio.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio processing method, device, electronic device and storage medium. Background Art

[0002] At present, in the application of audio transmission, the encoder encodes the original audio and transmits it. The audio decoded by the decoder is distorted compared with the original audio. In particular, when the encoder encodes the original audio at a very low bit rate, the original audio is distorted more seriously, the low-frequency part of the audio is damaged and the high-frequency part of the audio is lost, making the audio quality worse, thus affecting the user's sound quality experience. Therefore, how to improve the sound quality of the decoded audio and thus improve the user's sound quality experience is an urgent problem to be solved. Summary of the invention

[0003] In response to the above technical problems, the embodiments of the present application provide an audio processing method, device, electronic device and storage medium, which improve the sound quality of the audio to be processed by repairing the low-frequency information of the audio to be processed and compensating for the high-frequency information of the audio to be processed, thereby improving the user's sound quality experience.

[0004] In a first aspect, an embodiment of the present application provides an audio processing method, the method comprising:

[0005] Get the audio to be processed;

[0006] Divide the audio to be processed into frames to obtain multiple audio frames;

[0007] Performing frequency domain transformation on each audio frame to obtain a first spectrogram and a second spectrogram of each audio frame;

[0008] According to the encoding parameters of the audio to be processed, a first reference parameter and a second reference parameter are obtained, wherein the first reference parameter is used to repair the low frequency of each audio frame, and the second reference parameter is used to compensate the high frequency of each audio frame;

[0009] According to the first spectrogram of each audio frame and the first reference parameter, the low frequency of each audio frame is repaired to obtain a third spectrogram of each audio frame;

[0010] Compensate the high frequency of each audio frame according to the first spectrum graph, the second spectrum graph, and the second reference parameter of each audio frame to obtain a fourth spectrum graph of each audio frame;

[0011] The third spectrogram and the fourth spectrogram of each audio frame are fused to obtain a target spectrogram of each audio frame;

[0012] A time domain transform is performed on a plurality of target spectrograms corresponding to a plurality of audio frames to obtain a target audio.

[0013] In a second aspect, an embodiment of the present application provides an audio processing device, the device comprising: an acquisition unit and a processing unit;

[0014] An acquisition unit, used for acquiring audio to be processed;

[0015] A processing unit, used for dividing the audio to be processed into frames to obtain multiple audio frames;

[0016] A processing unit, configured to perform frequency domain transformation on each audio frame to obtain a first spectrum graph and a second spectrum graph of each audio frame;

[0017] A processing unit, configured to obtain a first reference parameter and a second reference parameter according to encoding parameters of the audio to be processed, wherein the first reference parameter is used to repair the low frequency of each audio frame, and the second reference parameter is used to compensate the high frequency of each audio frame;

[0018] A processing unit, configured to repair the low frequency of each audio frame according to the first spectrogram of each audio frame and the first reference parameter to obtain a third spectrogram of each audio frame;

[0019] A processing unit, configured to compensate the high frequency of each audio frame according to the first spectrogram, the second spectrogram, and the second reference parameter of each audio frame to obtain a fourth spectrogram of each audio frame;

[0020] A processing unit, configured to fuse the third spectrogram and the fourth spectrogram of each audio frame to obtain a target spectrogram of each audio frame;

[0021] The processing unit is used to perform time domain transformation on multiple target spectrograms corresponding to multiple audio frames to obtain target audio.

[0022] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, the processor being connected to the memory, the memory being used to store a computer program, and the processor being used to execute the computer program stored in the memory, so that the electronic device executes the method of the first aspect.

[0023] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program enables a computer to execute the method of the first aspect.

[0024] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and is computer-operable to cause the computer to execute the method of the first aspect.

[0025] In implementing the embodiment of the present application, the audio to be processed is divided into frames to obtain multiple audio frames; each audio frame is then transformed in the frequency domain to obtain a first spectrum map and a second spectrum map of each audio frame; then, according to the encoding parameters of the audio to be processed, the first reference parameter and the second reference parameter are obtained; then, according to the first spectrum map and the first reference parameter of each audio frame, the low frequency of each audio frame is repaired to obtain a third spectrum map of each audio frame; and according to the first spectrum map, the second spectrum map and the second reference parameter of each audio frame, the high frequency of each audio frame is compensated to obtain a fourth spectrum map of each audio frame; then, the third spectrum map and the fourth spectrum map of each audio frame are fused to obtain a target spectrum of each audio frame; finally, the target spectrum map of each audio frame is transformed in the time domain to obtain the target audio, the low frequency information of the audio to be processed is repaired and the high frequency information of the audio to be processed is compensated, the sound quality of the audio to be processed is improved, and the sound quality experience of the user is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 A schematic diagram of an audio processing system provided in an embodiment of the present application;

[0028] Figure 2 A flowchart of an audio processing method provided in an embodiment of the present application;

[0029] Figure 3 A schematic diagram of a first spectrum diagram of an i-th audio frame provided in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of a first spectrum diagram of a k-th audio frame provided in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of a second spectrum diagram of a k-th audio frame provided in an embodiment of the present application;

[0032] Figure 6 A block diagram of the functional units of an audio processing device provided in an embodiment of the present application;

[0033] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] The terms "first", "second", "third" and "fourth" etc. in the specification, claims and drawings of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0036] Reference to "embodiments" herein means that a particular feature, result, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0037] See also Figure 1 , Figure 1 A schematic diagram of an audio processing system provided for an embodiment of the present application, the system includes an encoding end 101 and a decoding end 102; wherein the encoding end 101 can be a user device, a streaming media platform, a computer device, etc.; the decoding end 102 can also be a user device, a streaming media platform, a computer device, etc.; the present application does not limit the encoding end 101 and the decoding end 102.

[0038] The encoding end 101 encodes the first audio to be transmitted to obtain the first digital audio. It should be noted that the encoding parameters for encoding the first audio in the present application are not limited. For example, the audio can be encoded at a very low bit rate to improve the transmission efficiency of the first digital audio and save costs; then the encoding end 101 sends the first digital audio to the decoding end 102; then the decoding end 102 obtains the first digital audio from the encoding end 101, decodes the first digital audio, and obtains the audio to be processed; then the decoding end 102 processes the audio to be processed according to the audio processing method of the present application to improve the sound quality of the audio to be processed and obtain the target audio.

[0039] Specifically, the decoding end 102 divides the audio to be processed into frames to obtain multiple audio frames; then the decoding end 102 performs frequency domain transformation on each audio frame to obtain a first spectrum map and a second spectrum map of each audio frame; in addition, the decoding end 102 obtains a first reference parameter and a second reference parameter according to the encoding parameters of the audio to be processed, wherein the first reference parameter is used to repair the low frequency of each audio frame, and the second reference parameter is used to compensate for the high frequency of each audio frame; then the decoding end 102 repairs the low frequency of each audio frame according to the first spectrum map and the first reference parameter of each audio frame to obtain a third spectrum map of each audio frame; the decoding end 102 also compensates for the high frequency of each audio frame according to the first spectrum map, the second spectrum map and the second reference parameter of each audio frame to obtain a fourth spectrum map of each audio frame; then the decoding end 102 merges the third spectrum map and the fourth spectrum map of each audio frame to obtain a target audio map of each audio frame; then, the decoding end performs time domain transformation on multiple target spectrum maps corresponding to multiple audio frames to obtain target audio.

[0040] For ease of understanding, this application will illustrate the specific application scenarios of the above audio processing method by examples, as follows:

[0041] First, when the user wants to upload audio A to the streaming platform, the encoding end 101 can be understood as the user device corresponding to the user, and the decoding end 102 can be understood as the streaming platform. Specifically, the user device encodes audio A to obtain digital audio A, and then uploads digital audio A to the streaming platform; then the streaming platform decodes the received digital audio A to obtain audio B; then the streaming platform repairs the low frequency of audio B according to the audio processing method of the present application, and compensates for the high frequency of audio B to obtain the target audio. Since in the process of encoding and transmitting audio A, audio A will be distorted, that is, audio B will be compared with audio A, such as low-frequency or high-frequency information loss, sound quality reduction and other problems, therefore, the streaming platform processes audio B according to the method of the present application, to repair the low frequency of audio B, and compensate for the high frequency of audio B, and then restore the sound quality of audio B, that is, to obtain the target audio, so that the target audio is as close to audio A as possible, to ensure the quality of audio A uploaded by the user, and to improve the user's sound quality experience.

[0042] Second, when the user wants to download and play audio C from the streaming platform, the encoding end 101 can be understood as the streaming platform, and the decoding end 102 can be understood as the user device. Specifically, the streaming platform encodes the audio C that the user wants to download to obtain digital audio B; then the streaming platform sends the digital audio B to the user device corresponding to the user; after the user device receives the digital audio B, the digital audio B is decoded to obtain audio D; then the user device repairs the low frequency of audio D according to the audio processing method of this application, and compensates for the high frequency of audio D to obtain the target audio. Similarly, since audio C will be distorted in the process of encoding and transmitting audio C, that is, audio D will be compared with audio C, such as low-frequency or high-frequency information loss, reduced sound quality and other problems, therefore, the user device processes audio D according to the method of this application to repair the low frequency of audio D and compensate for the high frequency of audio D, thereby restoring the sound quality of audio D, that is, obtaining the target audio, so that the target audio is as close to audio C as possible, ensuring the quality of audio C downloaded and played by the user, and improving the user's sound quality experience.

[0043] It can be seen that in the embodiment of the present application, a plurality of audio frames are obtained by dividing the audio to be processed into frames; each audio frame is then transformed in the frequency domain to obtain a first spectrum map and a second spectrum map of each audio frame; then, according to the encoding parameters of the audio to be processed, a first reference parameter and a second reference parameter are obtained; then, according to the first spectrum map and the first reference parameter of each audio frame, the low frequency of each audio frame is repaired to obtain a third spectrum map of each audio frame; and according to the first spectrum map, the second spectrum map and the second reference parameter of each audio frame, the high frequency of each audio frame is compensated to obtain a fourth spectrum map of each audio frame; then, the third spectrum map and the fourth spectrum map of each audio frame are fused to obtain a target spectrum of each audio frame; finally, the target spectrum map of each audio frame is transformed in the time domain to obtain the target audio, the low frequency information of the audio to be processed is repaired and the high frequency information of the audio to be processed is compensated, the sound quality of the audio to be processed is improved, and thus the sound quality experience of the user is improved.

[0044] See also Figure 2 , Figure 2 A flowchart of an audio processing method provided in an embodiment of the present application, the method is applied to Figure 1 The audio processing system shown includes but is not limited to steps 201-208:

[0045] 201: Get the audio frame to be processed.

[0046] It should be noted that after the encoding end encodes the first audio to be transmitted, the first digital audio is obtained; then the encoding end sends the first digital audio to the decoding end; and after the decoding end decodes the first digital audio, the audio to be processed is obtained.

[0047] 202: Divide the audio to be processed into frames to obtain multiple audio frames.

[0048] 203: Perform frequency domain transformation on each audio frame to obtain a first spectrum graph and a second spectrum graph of each audio frame.

[0049] Exemplarily, a frequency domain transform is performed on each audio frame, that is, a Fourier transform is performed on each audio frame to obtain a first spectrum diagram and a second spectrum diagram of each audio frame; wherein the first spectrum diagram can be an amplitude diagram, that is, based on the first spectrum diagram of each audio frame, the amplitude of each frequency point corresponding to each audio frame can be known; the second spectrum diagram can be a phase diagram, that is, based on the second spectrum diagram of each audio frame, the phase of each frequency point corresponding to each audio frame can be known.

[0050] 204: Obtain a first reference parameter and a second reference parameter according to encoding parameters of the audio to be processed.

[0051] The encoding parameters include one or more of the bit rate, number of channels and sampling rate of the audio to be processed, and the bit rate is less than a preset threshold. For example, by encoding the audio at a lower bit rate, costs can be saved and the transmission efficiency of the audio can be accelerated. In addition, the first reference parameter is used to repair the low frequency of each audio frame, and the second reference parameter is used to compensate for the high frequency of each audio frame.

[0052] It should be noted that before obtaining the first reference parameter and the second reference parameter according to the encoding parameters of the audio frame to be processed, the first training model can be trained based on the deep learning algorithm to obtain the reference parameters of the first target model, and then the first reference parameters corresponding to the first target model can be obtained according to the encoding parameters of the audio frame to be processed and the first target model, that is, the audio to be processed is input into the first target model, and then the first reference parameters are obtained according to the encoding parameters of the audio to be processed, so that the subsequent steps can obtain the third spectrum map of each audio frame based on the first reference parameters and the first spectrum map of each audio frame. Similarly, the second training model can also be trained based on the deep learning algorithm to obtain the reference parameters of the second target model, and then the second reference parameters corresponding to the first target model can be obtained according to the encoding parameters of the audio frame to be processed and the second target model, that is, the audio frame to be processed is input into the first audio model, and then the second reference parameters are obtained according to the encoding parameters of the audio to be processed, so that the subsequent steps can obtain the third spectrum map of each audio frame based on the second reference parameters and the second spectrum map of each audio frame. It should be noted that the specific method of training the first training model to obtain the reference parameters of the first target model and the specific method of training the second training model to obtain the reference parameters of the second target model are described in detail below and are not described here.

[0053] 205: Repair the low frequency of each audio frame according to the first spectrogram of each audio frame and the first reference parameter to obtain a third spectrogram of each audio frame.

[0054] In one embodiment, step 205 includes but is not limited to steps S11-S13:

[0055] S11: For the ith audio frame, a first feature matrix of the ith audio frame is obtained according to the first spectrogram of the ith audio frame, the m first spectrograms corresponding to the first m audio frames of the ith audio frame, and the n first spectrograms corresponding to the last n audio frames of the ith audio frame, wherein the ith audio frame is any one of the multiple audio frames.

[0056] In one embodiment, step S11 can be specifically as follows: according to the first eigenvalues ​​of each frequency point in the first spectrum graph of each audio frame in the i-th audio frame, the first m audio frames, and the last n audio frames, obtain m+n+1 first spectrum vectors corresponding to m+n+1 audio frames, wherein each first spectrum vector is composed of the first eigenvalues ​​of each frequency point in the first spectrum graph of each audio frame in the m+n+1 audio frames; then combine the m+n+1 first spectrum vectors to obtain the first characteristic matrix of the i-th audio frame; it should be noted that the present application does not limit the specific values ​​of m and n.

[0057] For example, see Figure 3 , Figure 3 A schematic diagram of a first spectrum diagram of an i-th audio frame provided in an embodiment of the present application, 1≤i≤Q, Q is the number of multiple audio frames, f1, f2, f3…fr are the frequencies of the i-th audio frame in the frequency domain, r is the number of frequencies of the i-th audio frame in the frequency domain, Mir is the first eigenvalue of the fr-th frequency point in the first spectrum diagram of the i-th audio frame, that is, the amplitude, then the first spectrum vector corresponding to the i-th audio frame is [Mi1, Mi2, Mi3, …, Mir], that is, the first spectrum vector is a row vector, or [Mi1, Mi2, Mi3, …, Mir] T , that is, the first spectrum vector is a column vector. Similarly, m first spectrum vectors corresponding to the first m audio frames of the i-th audio frame and n first spectrum vectors corresponding to the last n audio frames of the i-th audio frame are obtained; then the m+n+1 first spectrum vectors which are row vectors are combined to obtain the first characteristic matrix of the i-th audio frame. In other words, the above m+n+1 first spectrum vectors which are row vectors constitute the elements in the first characteristic matrix of the i-th audio frame; wherein, the first characteristic matrix of the i-th audio frame is:

[0058]

[0059] Optionally, m+n+1 first spectrum vectors that are column vectors may be combined to obtain a first feature matrix of the i-th audio frame. That is, the m+n+1 first spectrum vectors that are column vectors constitute elements in the first feature matrix of the i-th audio frame; wherein the first feature matrix of the i-th audio frame is:

[0060]

[0061] It should be noted that when repairing the low-frequency information of the processed audio, the low-frequency information of each audio frame of the processed audio is repaired; then, when repairing the low-frequency information of each audio frame, for example, the i-th audio frame, it is necessary to combine the first spectrum graph of the i-th audio frame and the m first spectrum graphs of the m audio frames before the i-th audio frame and the n first spectrum graphs of the n audio frames after the i-th audio frame to repair the low-frequency information of the i-th audio frame, thereby increasing the repair accuracy of the i-th audio frame.

[0062] In addition, when m≥i-1, the first spectrum vectors corresponding to (m-i+1) audio frames need to be padded with zeros, and when n≥Qi, the first spectrum vectors corresponding to (n+iQ) audio frames need to be padded with zeros.

[0063] S12: Obtaining restoration coefficients corresponding to each frequency point in a first spectrum graph of each audio frame according to the first feature matrix and the first reference parameter of each audio frame.

[0064] Exemplarily, the first feature matrix of each audio frame is input into the first target model, and then the first reference parameter of the first target model is called according to the encoding parameter of the audio to be processed. Then, the first target model obtains the restoration coefficients corresponding to each frequency point in the first spectrum graph of each audio frame based on the first reference parameter and the first feature matrix.

[0065] S13: based on the restoration coefficients corresponding to the respective frequency points in the first spectrum graph of each audio frame, the first eigenvalues ​​corresponding to the respective frequency points in the first spectrum graph of each audio frame are restored to obtain a third spectrum graph of each audio frame.

[0066] Exemplarily, the first amplitude of each frequency point corresponding to each audio frame can be obtained based on the product of the restoration coefficient corresponding to each frequency point in the first spectrum graph of each audio frame and the first eigenvalue (i.e., amplitude) corresponding to each frequency point in the first spectrum graph of each audio frame, and then the third spectrum graph of each audio frame can be obtained based on the first amplitude of each frequency point corresponding to each audio frame, that is, the third spectrum graph is an amplitude spectrum graph constituted by the first amplitude obtained after repairing the low frequency of each audio frame.

[0067] It should be noted that when repairing the low-frequency information of the audio frame, since the phase of each frequency point in the low-frequency information generally does not shift or change when the audio is encoded using a low bit rate, for the low-frequency information of each audio frame, only the amplitude needs to be repaired, and there is no need to adjust the phase in the low-frequency information. Of course, the above only uses the case where the first eigenvalue is the amplitude as an example, but the first eigenvalue corresponding to each frequency point in the first spectrum diagram of each audio frame may include power, energy, etc. in addition to the amplitude, and this application does not limit this.

[0068] 206: Compensate for the high frequency of each audio frame according to the first spectrum graph, the second spectrum graph, and the second reference parameter of each audio frame to obtain a fourth spectrum graph of each audio frame.

[0069] In one implementation, the fourth spectrogram of each audio frame includes a fifth spectrogram and a sixth spectrogram, and step 206 includes but is not limited to steps S21-S24:

[0070] S21: For the kth audio frame, a second feature matrix of the kth audio frame is obtained according to the first spectrogram and the second spectrogram of the kth audio frame, the x first spectrograms and the x second spectrograms corresponding to the first x audio frames of the kth audio frame, and the y first spectrograms and the y second spectrograms corresponding to the last y audio frames of the kth audio frame, wherein the kth audio frame is any one of the multiple audio frames. It should be noted that the present application does not limit the specific values ​​of x and y.

[0071] In one implementation, step S21 may be:

[0072] According to the first eigenvalue of each frequency point in the first spectrum graph of each audio frame in the kth audio frame, the first x audio frames and the last y audio frames, and the fourth eigenvalue of each frequency point in the second spectrum graph, the conjugate expression of each frequency point corresponding to each audio frame in the x+y+1 audio frames is obtained; then, according to the conjugate expression of each frequency point corresponding to each audio frame in the x+y+1 audio frames, x+y+1 second spectrum vectors corresponding to the x+y+1 audio frames are obtained, wherein each second spectrum vector is composed of the conjugate expression of each frequency point corresponding to each audio frame in the x+y+1 audio frames; then, the x+y+1 second spectrum vectors are combined to obtain a second characteristic matrix of the kth audio frame.

[0073] For example, see Figure 4 , Figure 4 A schematic diagram of a first spectrum diagram of a k-th audio frame provided by an embodiment of the present application, 1≤k≤Q, Q is the number of multiple audio frames, f1, f2, f3...fu are the frequency points of the k-th audio frame in the frequency domain, u is the number of frequency points of the k-th audio frame in the frequency domain, and Mku is the first eigenvalue, i.e., amplitude, of the fu-th frequency point in the first spectrum diagram of the k-th audio frame. Figure 5 , Figure 5 A schematic diagram of a second spectrum diagram of a k-th audio frame provided in an embodiment of the present application, f1, f2, f3…fu are the frequency points of the k-th audio frame in the frequency domain, u is the number of frequency points of the k-th audio frame in the frequency domain, and Nku is the fourth eigenvalue, i.e., phase, of the fu-th frequency point in the second spectrum diagram of the k-th audio frame. Then, according to the first eigenvalue (i.e., amplitude) of each frequency point in the first spectrum diagram of the k-th audio frame and the fourth eigenvalue (i.e., phase) of each frequency point in the second spectrum diagram of the k-th audio frame, the amplitude and phase of each frequency point corresponding to the k-th audio frame are obtained, and then based on the amplitude and phase of each frequency point corresponding to the k-th audio frame, the conjugate expression of each frequency point corresponding to the k-th audio frame is obtained.

[0074] Then, according to the conjugate expression of each frequency point corresponding to the k-th audio frame, the second spectrum vector corresponding to the k-th audio frame is obtained, that is, the conjugate expression of each frequency point corresponding to the k-th audio frame constitutes the elements of the second spectrum vector, wherein the second spectrum vector can be a row vector or a column vector, and the present application does not limit the form of the second spectrum vector. Similarly, x second spectrum vectors corresponding to the first x audio frames and y second spectrum vectors corresponding to the last y audio frames can be obtained.

[0075] Further, the second spectrum vector corresponding to the kth audio frame, the x second spectrum vectors corresponding to the first x audio frames of the kth audio frame, and the y second spectrum vectors corresponding to the last y audio frames of the kth audio frame are combined, that is, the x+y+1 second spectrum vectors are combined to obtain the second characteristic matrix of the kth audio frame, that is, the x+y+1 second spectrum vectors constitute the elements in the second characteristic matrix of the kth audio frame, wherein the method of combining the x+y+1 second spectrum vectors to obtain the second characteristic matrix can refer to the above-mentioned method of combining m+n+1 first spectrum vectors that are row vectors or column vectors to obtain the first characteristic matrix, and will not be repeated here.

[0076] It should be noted that when repairing the high-frequency information of the processed audio, the high-frequency information of each audio frame of the processed audio is compensated; then, when compensating the high-frequency information of each audio frame, such as the kth audio frame, it is necessary to combine the first spectrum graph and the second spectrum graph of the kth audio frame, the x first spectrum graphs and the x second spectrum graphs corresponding to the first x audio frames of the kth audio frame, and the y first spectrum graphs and the y second spectrum graphs corresponding to the last y audio frames of the kth audio frame, to compensate for the high-frequency information of the kth audio frame, which can increase the compensation accuracy of the kth audio frame. Of course, the above only takes the case where the first eigenvalue is the amplitude and the second eigenvalue is the phase as an example, but the first eigenvalue of each frequency point in the first spectrum graph of each audio frame mentioned above can include power, energy, etc. in addition to the amplitude, and the fourth eigenvalue of each frequency point in the second spectrum graph can also retain power, energy, etc. in addition to the phase, and the present application does not limit it.

[0077] In addition, when x≥k-1, the second spectrum vectors corresponding to (x-k+1) audio frames need to be padded with zeros, and when y≥Qk, the second spectrum vectors corresponding to (y+kQ) audio frames need to be padded with zeros.

[0078] S22: Obtaining a second eigenvalue and a third eigenvalue of a high frequency point of each audio frame according to the second eigenmatrix and the second reference parameter of each audio frame.

[0079] Exemplarily, the second characteristic matrix of each audio frame is input into the second target model, and then the second reference parameters of the second target model are called according to the encoding parameters of the audio to be processed. Then, the second target model obtains the second eigenvalue and the third eigenvalue of the high-frequency point of each audio frame based on the second reference parameters and the second characteristic matrix.

[0080] It should be noted that the number of high-frequency points in each audio frame depends on the second reference parameter, that is, the second reference parameter determines the number of high-frequency points obtained after compensation of each audio frame.

[0081] S23: Obtain a fifth frequency spectrum diagram of each audio frame according to the second eigenvalue of the high frequency point of each audio frame.

[0082] Exemplarily, the second eigenvalue of the high-frequency point of each audio frame can be the first phase, and then based on the first phase of the high-frequency point of each audio frame, the fifth spectrum diagram of each audio frame is obtained, that is, the fifth spectrum diagram is a phase spectrum diagram constructed by the first phase obtained after compensating for the high frequency of each audio frame.

[0083] S24: Obtain a sixth frequency spectrum of each audio frame according to the third eigenvalue of the high-frequency point of each audio frame.

[0084] Exemplarily, the third eigenvalue of the high-frequency frequency point of each audio frame can be the second amplitude, and then based on the second amplitude of the high-frequency frequency point of each audio frame, the sixth spectrum diagram of each audio frame is obtained, that is, the sixth spectrum diagram is an amplitude spectrum diagram composed of the second amplitude obtained after compensating the high frequency of each audio frame.

[0085] 207: Fusing the third spectrogram and the fourth spectrogram of each audio frame to obtain a target spectrogram of each audio frame.

[0086] Among them, the target spectrogram of each audio frame includes a seventh spectrogram and an eighth spectrogram. Exemplarily, the fourth eigenvalue (i.e., phase) of A1 frequency points in the second spectrogram of each audio frame and the second eigenvalue (i.e., first phase) of A2 frequency points in the fifth spectrogram of each audio frame are determined as the fifth eigenvalue of each frequency point in the seventh spectrogram of each audio frame, wherein the fifth eigenvalue represents the second phase, the seventh spectrogram is a phase spectrogram, and the number of each frequency point in the seventh spectrogram of each audio frame is the sum of the number of A1 frequency points in the first spectrogram of each audio frame and the number of A2 frequency points in the fifth spectrogram of each audio frame. In other words, the phase of each audio frame before low-frequency repair is used as the second phase of the low-frequency part of the audio frame, and the phase of each audio frame after high-frequency compensation is used as the second phase of the high-frequency part of the audio frame, so as to obtain the fifth eigenvalue (i.e., second phase) of each frequency point in the seventh spectrogram of each audio frame.

[0087] Then, the sixth eigenvalue (i.e., first amplitude) of A3 frequency points in the third spectrum graph of each audio frame and the third eigenvalue (i.e., second amplitude) of A4 frequency points in the sixth spectrum graph of each audio frame are determined as the seventh eigenvalue of each frequency point in the eighth spectrum graph of each audio frame, wherein the seventh eigenvalue is the third amplitude, the eighth spectrum graph is the amplitude spectrum graph, and the number of each frequency point in the eighth spectrum graph of each audio frame is the sum of the number of A3 frequency points in the third spectrum graph of each audio frame and the number of A4 frequency points in the sixth spectrum graph of each audio frame. In other words, the first amplitude after low-frequency repair of each audio frame is used as the third amplitude of the low-frequency part of the audio frame, and the second amplitude after high-frequency compensation of each audio frame is used as the third amplitude of the high-frequency part of the audio frame, so as to obtain the seventh eigenvalue (i.e., third amplitude) of each frequency point in the eighth spectrum graph of each audio frame.

[0088] 208: Perform time domain transformation on multiple target spectrograms corresponding to multiple audio frames to obtain target audio.

[0089] Exemplarily, a plurality of target spectrograms corresponding to a plurality of audio frames are transformed in the time domain to obtain a plurality of candidate audio frames; and then the plurality of candidate audio frames are concatenated to obtain the target audio.

[0090] In one implementation, before obtaining the first reference parameter and the second reference parameter according to the encoding parameters of the audio to be processed, the present application provides a method for training a first training model based on a first training audio sample to obtain reference parameters of a first target model, specifically comprising:

[0091] First, a first training audio sample is obtained, wherein the first training audio sample includes low-frequency information and high-frequency information; then, the first training audio sample is encoded using a first encoding parameter to obtain a first audio signal sample; then, the first audio signal sample is decoded to obtain a second training audio sample. The first encoding parameter is not limited in this application.

[0092] Then, the second training audio sample is divided into frames to obtain a plurality of first training audio frames; and the first training audio sample is divided into frames to obtain a plurality of first training audio frame labels.

[0093] Then, according to the multiple first training audio frames and the multiple first training audio frame labels, the initial parameters of the first training model are trained to obtain the reference parameters of the first target model, wherein the reference parameters of the first target model correspond to the first encoding parameters. Exemplarily, each first training audio frame is transformed in the frequency domain to obtain the first spectrogram of each first training audio frame; then, for the first training audio frame A, the first spectrogram of the first training audio frame A, the m first spectrograms corresponding to the first m first training audio frames of the first training audio frame A, and the n first spectrograms corresponding to the last n first training audio frames of the first training audio frame A are input into the first training model to obtain the restoration coefficients corresponding to each frequency point in the first spectrogram of the first training audio frame A, wherein the first training audio frame A is any one of the multiple first training audio frames.

[0094] Then, based on the restoration coefficients corresponding to each frequency point in the first spectrum graph of each first training audio frame and multiple first training audio frame labels, the initial parameters of the first training model are trained to obtain the reference parameters of the first target model. Exemplarily, based on the restoration coefficients corresponding to each frequency point in the first spectrum graph of each first training audio frame, the low frequency of each first training audio frame is restored to obtain multiple second training audio frames; then, according to the characteristic values ​​of each frequency point corresponding to each second training audio frame (here, the characteristic value is an example of amplitude), and the characteristic values ​​(i.e., amplitude) of each frequency point corresponding to the first training audio frame label corresponding to each second training audio frame, the first loss value of each frequency point corresponding to each second training audio frame is obtained. Then, based on the first loss value of each frequency point corresponding to each second training audio frame, the second loss value of each second training audio frame is obtained, for example, the second loss value of each second training audio frame can be the weighted average of the first loss values ​​of each frequency point corresponding to each second training audio frame. Then, according to the second loss value of each second training audio frame, the initial parameters of the first training model are trained to obtain reference parameters of the first target model. That is to say, based on the second loss value of each second training audio frame, the initial parameters of the first training model are adjusted to obtain the reference parameters of the first target model, wherein the obtained reference parameters correspond to the first encoding parameters for encoding the first training audio sample.

[0095] Similarly, in one embodiment, before obtaining the first reference parameter and the second reference parameter according to the encoding parameter of the audio to be processed, the present application also provides a method for training a second training model based on a second training audio sample to obtain reference parameters of a second target model, specifically comprising:

[0096] First, a second training audio sample is obtained, wherein the second training audio sample includes high-frequency information and low-frequency information; then, the second training audio sample is encoded using a second encoding parameter to obtain a second audio signal sample; then, the second audio signal sample is decoded to obtain a third training audio sample; it should be noted that the present application does not limit the second encoding parameter.

[0097] Then, the third training audio sample is divided into frames to obtain a plurality of third training audio frames; and the second training audio sample is divided into frames to obtain a plurality of third training audio frame labels.

[0098] Then, according to the plurality of third training audio frames and the plurality of third training audio frame labels, the initial parameters of the second training model are trained to obtain reference parameters of the second target model, wherein the reference parameters of the second target model correspond to the second encoding parameters. Exemplarily, each third training audio frame is transformed in the frequency domain to obtain the first spectrogram and the second spectrogram of each third training audio frame; then, for the third training audio frame A, the first spectrogram and the second spectrogram of the third training audio frame A, the x first spectrograms and the x second spectrograms corresponding to the first x third training audio frames of the third training audio frame A, and the y first spectrograms and the y second spectrograms corresponding to the last y third training audio frames of the third training audio frame A are input into the second training model to obtain the second eigenvalue (i.e., the first phase) and the third eigenvalue (i.e., the second amplitude) of the high frequency point of the third training audio frame A, wherein the third training audio frame A is any one of the plurality of third training audio frames.

[0099] Then, based on the second eigenvalue and the third eigenvalue of the high frequency point of each third training audio frame, and multiple third training audio frame labels, the initial parameters of the second training model are trained to obtain the reference parameters of the second target model. Exemplarily, based on the second eigenvalue and the third eigenvalue of the high frequency point of each third training audio frame, and the second eigenvalue and the third eigenvalue of the high frequency point of the third training audio frame label corresponding to each third training audio frame, the third loss value of the high frequency point of each third training audio frame is obtained; then, based on the third loss value of the high frequency point of each third training audio frame, the fourth loss value of each third training audio frame is obtained, for example, the fourth loss value of each third training audio frame can be the weighted average of the third loss value of the high frequency point of each third training audio frame; then, based on the fourth loss value of each third training audio frame, the initial parameters of the second training model are trained to obtain the reference parameters of the second target model, wherein the obtained reference parameters correspond to the second encoding parameters for encoding the second training audio samples.

[0100] It can be seen that in the embodiment of the present application, a plurality of audio frames are obtained by dividing the audio to be processed into frames; each audio frame is then transformed in the frequency domain to obtain a first spectrum map and a second spectrum map of each audio frame; then, according to the encoding parameters of the audio to be processed, a first reference parameter and a second reference parameter are obtained; then, according to the first spectrum map and the first reference parameter of each audio frame, the low frequency of each audio frame is repaired to obtain a third spectrum map of each audio frame; and according to the first spectrum map, the second spectrum map and the second reference parameter of each audio frame, the high frequency of each audio frame is compensated to obtain a fourth spectrum map of each audio frame; then, the third spectrum map and the fourth spectrum map of each audio frame are fused to obtain a target spectrum of each audio frame; finally, the target spectrum map of each audio frame is transformed in the time domain to obtain the target audio, the low frequency information of the audio to be processed is repaired and the high frequency information of the audio to be processed is compensated, the sound quality of the audio to be processed is improved, and thus the sound quality experience of the user is improved.

[0101] See also Figure 6 , Figure 6 This is a block diagram of the functional units of an audio processing device provided in an embodiment of the present application. The audio processing device 600 includes: an acquisition unit 601 and a processing unit 602;

[0102] An acquisition unit 601 is used to acquire audio to be processed;

[0103] The processing unit 602 is used to divide the audio to be processed into frames to obtain multiple audio frames; perform frequency domain transformation on each audio frame to obtain a first spectrogram and a second spectrogram of each audio frame; obtain a first reference parameter and a second reference parameter according to the encoding parameters of the audio to be processed, wherein the first reference parameter is used to repair the low frequency of each audio frame, and the second reference parameter is used to compensate for the high frequency of each audio frame; according to the first spectrogram and the first reference parameter of each audio frame, the low frequency of each audio frame is repaired to obtain a third spectrogram of each audio frame; according to the first spectrogram, the second spectrogram, and the second reference parameter of each audio frame, the high frequency of each audio frame is compensated to obtain a fourth spectrogram of each audio frame; the third spectrogram and the fourth spectrogram of each audio frame are merged to obtain a target spectrogram of each audio frame; and multiple target spectrograms corresponding to multiple audio frames are transformed in the time domain to obtain target audio.

[0104] In one embodiment of the present application, in terms of repairing the low frequency of each audio frame according to the first spectrogram and the first reference parameter of each audio frame to obtain a third spectrogram of each audio frame, the processing unit 602 is specifically configured to:

[0105] For an i-th audio frame, obtaining a first feature matrix of the i-th audio frame according to a first spectrogram of the i-th audio frame, m first spectrograms corresponding to m audio frames before the i-th audio frame, and n first spectrograms corresponding to n audio frames after the i-th audio frame, wherein the i-th audio frame is any one audio frame among the multiple audio frames;

[0106] Obtaining restoration coefficients corresponding to each frequency point in a first spectrum graph of each audio frame according to a first feature matrix and a first reference parameter of each audio frame;

[0107] Based on the restoration coefficients corresponding to the respective frequency points in the first spectrum graph of each audio frame, the first eigenvalues ​​corresponding to the respective frequency points in the first spectrum graph of each audio frame are restored to obtain a third spectrum graph of each audio frame.

[0108] In one embodiment of the present application, in terms of obtaining a first feature matrix of the i-th audio frame according to the first spectrogram of the i-th audio frame, the m first spectrograms corresponding to the first m audio frames of the i-th audio frame, and the n first spectrograms corresponding to the last n audio frames of the i-th audio frame, the processing unit 602 is specifically configured to:

[0109] According to the first eigenvalues ​​of each frequency point in the first spectrum graph of each audio frame in the i-th audio frame, the first m audio frames, and the last n audio frames, m+n+1 first spectrum vectors corresponding to the m+n+1 audio frames are obtained, wherein each first spectrum vector is composed of the first eigenvalues ​​of each frequency point in the first spectrum graph of each audio frame in the m+n+1 audio frames;

[0110] The m+n+1 first spectrum vectors are combined to obtain a first feature matrix of the i-th audio frame.

[0111] In one embodiment of the present application, the fourth spectrogram of each audio frame includes a fifth spectrogram and a sixth spectrogram. In terms of compensating the high frequency of each audio frame according to the first spectrogram and the second spectrogram of each audio frame and the second reference parameter to obtain the fourth spectrogram of each audio frame, the processing unit 602 is specifically configured to:

[0112] For a k-th audio frame, obtaining a second feature matrix of the k-th audio frame according to the first spectrogram and the second spectrogram of the k-th audio frame, the x first spectrograms and the x second spectrograms corresponding to the x audio frames before the k-th audio frame, and the y first spectrograms and the y second spectrograms corresponding to the y audio frames after the k-th audio frame, wherein the k-th audio frame is any one audio frame among the multiple audio frames;

[0113] According to the second characteristic matrix of each audio frame and the second reference parameter, a second eigenvalue and a third eigenvalue of the high frequency point of each audio frame are obtained;

[0114] Obtaining a fifth frequency spectrum of each audio frame according to the second eigenvalue of the high frequency point of each audio frame;

[0115] A sixth frequency spectrum diagram of each audio frame is obtained according to the third eigenvalue of the high frequency point of each audio frame.

[0116] In one embodiment of the present application, in terms of obtaining a second feature matrix of the kth audio frame according to the first spectrogram and the second spectrogram of the kth audio frame, the x first spectrograms and the x second spectrograms corresponding to the first x audio frames of the kth audio frame, and the y first spectrograms and the y second spectrograms corresponding to the last y audio frames of the kth audio frame, the processing unit 602 is specifically configured to:

[0117] According to the first eigenvalue of each frequency point in the first spectrum graph of each audio frame in the kth audio frame, the first x audio frames, and the last y audio frames and the fourth eigenvalue of each frequency point in the second spectrum graph, a conjugate expression of each frequency point corresponding to each audio frame in the x+y+1 audio frames is obtained;

[0118] According to the conjugate expression of each frequency point corresponding to each audio frame in the x+y+1 audio frames, obtain x+y+1 second spectrum vectors corresponding to the x+y+1 audio frames, wherein each second spectrum vector is composed of the conjugate expression of each frequency point corresponding to each audio frame in the x+y+1 audio frames;

[0119] The x+y+1 second spectrum vectors are combined to obtain a second feature matrix of the kth audio frame.

[0120] In one embodiment of the present application, the target spectrogram includes a seventh spectrogram and an eighth spectrogram. In fusing the third spectrogram and the fourth spectrogram of each audio frame to obtain the target spectrogram of each audio frame, the processing unit 602 is specifically configured to:

[0121] Determine the fourth eigenvalue of A1 frequency points in the second spectrogram of each audio frame and the second eigenvalue of A2 frequency points in the fifth spectrogram of each audio frame as the fifth eigenvalue of each frequency point in the seventh spectrogram of each audio frame, wherein the number of each frequency point in the seventh spectrogram of each audio frame is the sum of the number of A1 frequency points in the first spectrogram of each audio frame and the number of A2 frequency points in the fifth spectrogram of each audio frame;

[0122] The sixth eigenvalue of A3 frequency points in the third spectrum graph of each audio frame and the third eigenvalue of A4 frequency point in the sixth spectrum graph of each audio frame are determined as the seventh eigenvalue of each frequency point in the eighth spectrum graph of each audio frame, wherein the number of each frequency point in the eighth spectrum graph of each audio frame is the sum of the number of A3 frequency points in the third spectrum graph of each audio frame and the number of A4 frequency points in the sixth spectrum graph of each audio frame.

[0123] In one embodiment of the present application, in performing time domain transformation on multiple target spectrograms corresponding to multiple audio frames to obtain target audio, the processing unit 602 is specifically configured to:

[0124] Performing time domain transformation on multiple target spectrograms corresponding to multiple audio frames to obtain multiple candidate audio frames;

[0125] Multiple candidate audio frames are concatenated to obtain the target audio.

[0126] In one embodiment of the present application, before obtaining the first reference parameter and the second reference parameter according to the encoding parameter of the audio to be processed, the processing unit 602 is specifically configured to:

[0127] Obtaining a first training audio sample;

[0128] Encoding a first training audio sample using a first encoding parameter to obtain a first audio signal sample;

[0129] Decoding the first audio signal sample to obtain a second training audio sample;

[0130] Dividing the second training audio sample into frames to obtain a plurality of first training audio frames;

[0131] Divide the first training audio sample into frames to obtain a plurality of first training audio frame labels;

[0132] According to the plurality of first training audio frames and the plurality of first training audio frame labels, initial parameters of the first training model are trained to obtain reference parameters of the first target model, wherein the reference parameters of the first target model correspond to the first encoding parameters.

[0133] In one embodiment of the present application, in terms of training the initial parameters of the first training model according to the plurality of first training audio frames and the plurality of first training audio frame labels to obtain reference parameters of the first target model, the processing unit 602 is specifically configured to:

[0134] Performing frequency domain transformation on each first training audio frame to obtain a first spectrogram of each first training audio frame;

[0135] For a first training audio frame A, a first spectrogram of the first training audio frame A, m first spectrograms corresponding to the first m first training audio frames of the first training audio frame A, and n first spectrograms corresponding to the last n first training audio frames of the first training audio frame A are input into a first training model to obtain a restoration coefficient corresponding to each frequency point in the first spectrogram of the first training audio frame A, wherein the first training audio frame A is any one audio frame of the multiple first training audio frames;

[0136] Based on the restoration coefficients corresponding to the respective frequency points in the first spectrum graph of each first training audio frame and a plurality of first training audio frame labels, the initial parameters of the first training model are trained to obtain reference parameters of the first target model.

[0137] In one embodiment of the present application, in terms of training the initial parameters of the first training model based on the restoration coefficients corresponding to each frequency point in the first spectrum graph of each first training audio frame and multiple first training audio frame labels to obtain reference parameters of the first target model, the processing unit 602 is specifically used to:

[0138] Based on the restoration coefficients corresponding to the frequency points in the first spectrum graph of each first training audio frame, the low frequency of each first training audio frame is restored to obtain a plurality of second training audio frames;

[0139] Obtaining a first loss value for each frequency point corresponding to each second training audio frame according to the feature values ​​of each frequency point corresponding to each second training audio frame and the feature values ​​of each frequency point corresponding to the first training audio frame label corresponding to each second training audio frame;

[0140] Obtaining a second loss value for each second training audio frame based on the first loss values ​​of each frequency point corresponding to each second training audio frame;

[0141] According to the second loss value of each second training audio frame, the initial parameters of the first training model are trained to obtain reference parameters of the first target model.

[0142] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device 700 includes a transceiver 701, a processor 702 and a memory 703. They are connected via a bus 704. The memory 703 is used to store computer programs and data, and can transmit the data stored in the memory 703 to the processor 702.

[0143] The processor 702 is used to read the computer program in the memory 703 and perform the following operations:

[0144] Controlling the transceiver 701 to obtain the audio to be processed;

[0145] Divide the audio to be processed into frames to obtain multiple audio frames;

[0146] Performing frequency domain transformation on each audio frame to obtain a first spectrogram and a second spectrogram of each audio frame;

[0147] According to the encoding parameters of the audio to be processed, a first reference parameter and a second reference parameter are obtained, wherein the first reference parameter is used to repair the low frequency of each audio frame, and the second reference parameter is used to compensate the high frequency of each audio frame;

[0148] According to the first spectrogram of each audio frame and the first reference parameter, the low frequency of each audio frame is repaired to obtain a third spectrogram of each audio frame;

[0149] Compensate the high frequency of each audio frame according to the first spectrum graph, the second spectrum graph, and the second reference parameter of each audio frame to obtain a fourth spectrum graph of each audio frame;

[0150] The third spectrogram and the fourth spectrogram of each audio frame are fused to obtain a target spectrogram of each audio frame;

[0151] A time domain transform is performed on a plurality of target spectrograms corresponding to a plurality of audio frames to obtain a target audio.

[0152] Specifically, the transceiver 701 may be Figure 6 The acquisition unit 601 of the audio processing device 600 of the embodiment, the processor 702 may be Figure 6 The processing unit 602 of the audio processing device 600 of the embodiment.

[0153] It should be understood that the electronic devices in this application may include smart phones (such as Android phones, iOS phones, Windows Phone phones, etc.), tablet computers, PDAs, laptops, mobile Internet devices MID (Mobile Internet Devices, referred to as: MID) or wearable devices, etc. The above electronic devices are only examples, not exhaustive, and include but are not limited to the above electronic devices. In practical applications, the above electronic devices may also include: smart vehicle terminals, computer equipment, etc.

[0154] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor to implement part or all of the steps of any audio processing method recorded in the above method embodiments.

[0155] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any audio processing method recorded in the above method embodiments.

[0156] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0157] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0158] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0159] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software program module.

[0161] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0162] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.

[0163] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An audio processing method, characterized in that: The method comprises: Get the audio to be processed; Divide the audio to be processed into frames to obtain multiple audio frames; Performing frequency domain transformation on each audio frame to obtain a first spectrum diagram and a second spectrum diagram of each audio frame, wherein the first spectrum diagram is an amplitude diagram, and the second spectrum diagram is a phase diagram; According to the encoding parameters of the audio to be processed, obtaining a first reference parameter and a second reference parameter, wherein the first reference parameter is a reference parameter of a first target model corresponding to the encoding parameter, and the reference parameter of the first target model is obtained by training the initial parameters of the first training model based on the first encoding parameter and the first training audio sample, and the second reference parameter is a reference parameter of a second target model corresponding to the encoding parameter, and the reference parameter of the second target model is obtained by training the initial parameters of the second training model based on the second encoding parameter and the second training audio sample; According to the first spectrum graph of each audio frame and the first reference parameter, the low frequency of each audio frame is repaired to obtain the third spectrum graph of each audio frame, specifically comprising: for the ith audio frame, according to the first spectrum graph of the ith audio frame, m first spectrum graphs corresponding to the first m audio frames of the ith audio frame, and n first spectrum graphs corresponding to the last n audio frames of the ith audio frame, a first characteristic matrix of the ith audio frame is obtained, wherein the ith audio frame is any one of the multiple audio frames; according to the first characteristic matrix of each audio frame and the first reference parameter, a repair coefficient corresponding to each frequency point in the first spectrum graph of each audio frame is obtained; based on the repair coefficient corresponding to each frequency point in the first spectrum graph of each audio frame, the first eigenvalue corresponding to each frequency point in the first spectrum graph of each audio frame is repaired to obtain the third spectrum graph of each audio frame; According to the first spectrogram, the second spectrogram, and the second reference parameter of each audio frame, the high frequency of each audio frame is compensated to obtain a fourth spectrogram of each audio frame, and the fourth spectrogram of each audio frame includes a fifth spectrogram and a sixth spectrogram, specifically comprising: for the kth audio frame, according to the first spectrogram and the second spectrogram of the kth audio frame, the x first spectrograms and the x second spectrograms corresponding to the first x audio frames of the kth audio frame, and the y first spectrograms and the y second spectrograms corresponding to the last y audio frames of the kth audio frame, obtain a second characteristic matrix of the kth audio frame, wherein the kth audio frame is any one of the multiple audio frames; according to the second characteristic matrix of each audio frame and the second reference parameter, obtain a second eigenvalue and a third eigenvalue of the high frequency point of each audio frame; according to the second eigenvalue of the high frequency point of each audio frame, obtain a fifth spectrogram of each audio frame; according to the third eigenvalue of the high frequency point of each audio frame, obtain a sixth spectrogram of each audio frame; The third spectrogram and the fourth spectrogram of each audio frame are merged to obtain a target spectrogram of each audio frame; Perform time domain transformation on multiple target spectrograms corresponding to the multiple audio frames to obtain target audio.

2. The method according to claim 1, characterized in that The step of obtaining a first feature matrix of the i-th audio frame according to the first spectrogram of the i-th audio frame, the m first spectrograms corresponding to the first m audio frames of the i-th audio frame, and the n first spectrograms corresponding to the last n audio frames of the i-th audio frame comprises: According to the first eigenvalues ​​of each frequency point in the first spectrum graph of each audio frame in the i-th audio frame, the first m audio frames, and the last n audio frames, obtaining m+n+1 first spectrum vectors corresponding to the m+n+1 audio frames, wherein each first spectrum vector is composed of the first eigenvalues ​​of each frequency point in the first spectrum graph of each audio frame in the m+n+1 audio frames; The m+n+1 first frequency spectrum vectors are combined to obtain a first feature matrix of the i-th audio frame.

3. The method according to claim 1 or 2, characterized in that: The second feature matrix of the k-th audio frame is obtained according to the first spectrogram and the second spectrogram of the k-th audio frame, the x first spectrograms and the x second spectrograms corresponding to the first x audio frames of the k-th audio frame, and the y first spectrograms and the y second spectrograms corresponding to the last y audio frames of the k-th audio frame, comprising: Obtain a conjugate expression of each frequency point corresponding to each audio frame in the x+y+1 audio frames according to the first eigenvalue of each frequency point in the first spectrum graph of each audio frame in the kth audio frame, the first x audio frames, and the last y audio frames and the fourth eigenvalue of each frequency point in the second spectrum graph; According to the conjugate expression of each frequency point corresponding to each audio frame in the x+y+1 audio frames, obtain x+y+1 second spectrum vectors corresponding to the x+y+1 audio frames, wherein each second spectrum vector is composed of the conjugate expression of each frequency point corresponding to each audio frame in the x+y+1 audio frames; The x+y+1 second spectrum vectors are combined to obtain a second feature matrix of the k-th audio frame.

4. The method according to claim 1 or 2, characterized in that: The target spectrum graph includes a seventh spectrum graph and an eighth spectrum graph; The fusing the third spectrogram and the fourth spectrogram of each audio frame to obtain a target spectrogram of each audio frame includes: Determine the fourth eigenvalue of A1 frequency points in the second spectrum graph of each audio frame and the second eigenvalue of A2 frequency points in the fifth spectrum graph of each audio frame as the fifth eigenvalue of each frequency point in the seventh spectrum graph of each audio frame, wherein the number of each frequency point in the seventh spectrum graph of each audio frame is the sum of the number of A1 frequency points in the first spectrum graph of each audio frame and the number of A2 frequency points in the fifth spectrum graph of each audio frame; The sixth eigenvalue of A3 frequency points in the third spectrum graph of each audio frame and the third eigenvalue of A4 frequency points in the sixth spectrum graph of each audio frame are determined as the seventh eigenvalue of each frequency point in the eighth spectrum graph of each audio frame, wherein the number of each frequency point in the eighth spectrum graph of each audio frame is the sum of the number of A3 frequency points in the third spectrum graph of each audio frame and the number of A4 frequency points in the sixth spectrum graph of each audio frame.

5. The method according to claim 1 or 2, characterized in that: The step of performing time domain transformation on the multiple target spectrograms corresponding to the multiple audio frames to obtain the target audio includes: Performing time domain transformation on a plurality of target spectrograms corresponding to the plurality of audio frames to obtain a plurality of candidate audio frames; The multiple candidate audio frames are concatenated to obtain the target audio.

6. The method according to claim 1 or 2, characterized in that: The encoding parameters include one or more of a bit rate, a number of channels, and a sampling rate of the audio to be processed, and the bit rate is less than a preset threshold.

7. The method according to claim 1 or 2, characterized in that: Before acquiring the first reference parameter and the second reference parameter according to the encoding parameter of the audio to be processed, the method further includes: Obtaining a first training audio sample; Encoding the first training audio sample using a first encoding parameter to obtain a first audio signal sample; Decoding the first audio signal sample to obtain a second training audio sample; Dividing the second training audio sample into frames to obtain a plurality of first training audio frames; Divide the first training audio sample into frames to obtain a plurality of first training audio frame labels; According to the multiple first training audio frames and the multiple first training audio frame labels, initial parameters of the first training model are trained to obtain reference parameters of the first target model, wherein the reference parameters of the first target model correspond to the first encoding parameters.

8. The method according to claim 7, characterized in that The step of training the initial parameters of the first training model according to the plurality of first training audio frames and the plurality of first training audio frame labels to obtain reference parameters of the first target model includes: Performing frequency domain transformation on each first training audio frame to obtain a first spectrogram of each first training audio frame; For a first training audio frame A, a first spectrogram of the first training audio frame A, m first spectrograms corresponding to the first m first training audio frames of the first training audio frame A, and n first spectrograms corresponding to the last n first training audio frames of the first training audio frame A are input into the first training model to obtain restoration coefficients corresponding to each frequency point in the first spectrogram of the first training audio frame A, wherein the first training audio frame A is any one audio frame of the multiple first training audio frames; Based on the restoration coefficients corresponding to the respective frequency points in the first spectrum graph of each first training audio frame and the multiple first training audio frame labels, the initial parameters of the first training model are trained to obtain reference parameters of the first target model.

9. The method according to claim 8, characterized in that The training of the initial parameters of the first training model based on the restoration coefficients corresponding to the respective frequency points in the first spectrum graph of each first training audio frame and the multiple first training audio frame labels to obtain the reference parameters of the first target model includes: Based on the restoration coefficients corresponding to the frequency points in the first spectrum graph of each first training audio frame, the low frequency of each first training audio frame is restored to obtain a plurality of second training audio frames; Obtaining a first loss value for each frequency point corresponding to each second training audio frame according to the feature values ​​of each frequency point corresponding to each second training audio frame and the feature values ​​of each frequency point corresponding to the first training audio frame label corresponding to each second training audio frame; Obtaining a second loss value for each second training audio frame based on the first loss values ​​of each frequency point corresponding to each second training audio frame; According to the second loss value of each second training audio frame, the initial parameters of the first training model are trained to obtain reference parameters of the first target model.

10. The method according to claim 1 or 2, characterized in that: Before acquiring the first reference parameter and the second reference parameter according to the encoding parameter of the audio to be processed, the method further includes: Obtaining a second training audio sample; Encoding the second training audio sample using a second encoding parameter to obtain a second audio signal sample; Decoding the second audio signal sample to obtain a third training audio sample; Dividing the third training audio sample into frames to obtain a plurality of third training audio frames; Divide the second training audio sample into frames to obtain a plurality of third training audio frame labels; According to the multiple third training audio frames and the multiple third training audio frame labels, the initial parameters of the second training model are trained to obtain reference parameters of the second target model, wherein the reference parameters of the second target model correspond to the second encoding parameters.

11. The method according to claim 10, characterized in that The step of training the initial parameters of the second training model according to the plurality of third training audio frames and the plurality of third training audio frame labels to obtain reference parameters of the second target model includes: Performing frequency domain transformation on each third training audio frame to obtain a first spectrogram and a second spectrogram of each third training audio frame; For a third training audio frame A, inputting a first spectrogram and a second spectrogram of the third training audio frame A, x first spectrograms and x second spectrograms corresponding to the first x third training audio frames of the third training audio frame A, and y first spectrograms and y second spectrograms corresponding to the last y third training audio frames of the third training audio frame A into the second training model to obtain a second eigenvalue and a third eigenvalue of a high-frequency point of the third training audio frame A, wherein the third training audio frame A is any one audio frame of the multiple third training audio frames; Based on the second eigenvalue and the third eigenvalue of the high frequency point of each third training audio frame and the multiple third training audio frame labels, the initial parameters of the second training model are trained to obtain the reference parameters of the second target model.

12. The method according to claim 11, characterized in that The training of initial parameters of the second training model based on the second eigenvalue and the third eigenvalue of the high frequency point of each third training audio frame and the plurality of third training audio frame labels to obtain reference parameters of the second target model includes: Obtaining a third loss value of the high frequency point of each third training audio frame based on the second eigenvalue and the third eigenvalue of the high frequency point of each third training audio frame and the second eigenvalue and the third eigenvalue of the high frequency point of the third training audio frame label corresponding to each third training audio frame; Obtaining a fourth loss value for each third training audio frame based on the third loss value of the high frequency point of each third training audio frame; Based on the fourth loss value of each third training audio frame, the initial parameters of the second training model are trained to obtain reference parameters of the second target model.

13. An audio processing device, characterized in that: The device comprises: an acquisition unit and a processing unit; The acquisition unit is used to acquire the audio to be processed; The processing unit is used to divide the audio to be processed into frames to obtain multiple audio frames; The processing unit is used to perform frequency domain transformation on each audio frame to obtain a first spectrum diagram and a second spectrum diagram of each audio frame, wherein the first spectrum diagram is an amplitude diagram and the second spectrum diagram is a phase diagram; The processing unit is used to obtain a first reference parameter and a second reference parameter according to the encoding parameter of the audio to be processed, wherein the first reference parameter is a reference parameter of a first target model corresponding to the encoding parameter, and the reference parameter of the first target model is obtained by training the initial parameters of the first training model based on the first encoding parameter and the first training audio sample, and the second reference parameter is a reference parameter of a second target model corresponding to the encoding parameter, and the reference parameter of the second target model is obtained by training the initial parameters of the second training model based on the second encoding parameter and the second training audio sample; The processing unit is used to repair the low frequency of each audio frame according to the first spectrum diagram of each audio frame and the first reference parameter to obtain the third spectrum diagram of each audio frame, specifically comprising: for the ith audio frame, according to the first spectrum diagram of the ith audio frame, m first spectrum diagrams corresponding to the first m audio frames of the ith audio frame, and n first spectrum diagrams corresponding to the last n audio frames of the ith audio frame, obtain the first characteristic matrix of the ith audio frame, wherein the ith audio frame is any audio frame of the multiple audio frames; according to the first characteristic matrix of each audio frame and the first reference parameter, obtain the repair coefficient corresponding to each frequency point in the first spectrum diagram of each audio frame; based on the repair coefficient corresponding to each frequency point in the first spectrum diagram of each audio frame, repair the first eigenvalue corresponding to each frequency point in the first spectrum diagram of each audio frame to obtain the third spectrum diagram of each audio frame; The processing unit is used to compensate the high frequency of each audio frame according to the first spectrogram, the second spectrogram, and the second reference parameter of each audio frame to obtain a fourth spectrogram of each audio frame, wherein the fourth spectrogram of each audio frame includes a fifth spectrogram and a sixth spectrogram, specifically comprising: for a k-th audio frame, according to the first spectrogram and the second spectrogram of the k-th audio frame, x first spectrograms and x second spectrograms corresponding to the first x audio frames of the k-th audio frame, and y first spectrograms and y second spectrograms corresponding to the last y audio frames of the k-th audio frame, obtaining a second characteristic matrix of the k-th audio frame, wherein the k-th audio frame is any one of the multiple audio frames; according to the second characteristic matrix of each audio frame and the second reference parameter, obtaining a second eigenvalue and a third eigenvalue of a high frequency point of each audio frame; according to the second eigenvalue of the high frequency point of each audio frame, obtaining a fifth spectrogram of each audio frame; according to the third eigenvalue of the high frequency point of each audio frame, obtaining a sixth spectrogram of each audio frame; The processing unit is used to fuse the third spectrogram and the fourth spectrogram of each audio frame to obtain a target spectrogram of each audio frame; The processing unit is used to perform time domain transformation on the multiple target spectrograms corresponding to the multiple audio frames to obtain target audio.

14. An electronic device, characterized in that: include: A processor and a memory, the processor is connected to the memory, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method as described in any one of claims 1-12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 12.

Images