Audio repairing method and device, computer device and storage medium

Abstract

The application relates to an audio repairing method and device, computer equipment, a storage medium and a computer program product, which can be used in the technical field of audio processing and can also be used in the field of financial technology or other related fields. The method comprises the following steps: obtaining an original spectrum graph of an audio to be repaired of a financial system; performing feature extraction processing on the original spectrum graph to obtain an original spectrum feature graph of the audio to be repaired, performing down-sampling processing on the original spectrum feature graph at different multiples to obtain spectrum feature graphs of multiple scales of the audio to be repaired; performing feature extraction processing on the spectrum feature graphs of the multiple scales to obtain target spectrum feature graphs of the multiple scales; performing audio reconstruction processing according to the audio to be repaired and the target spectrum feature graphs of the multiple scales to obtain a target spectrum graph of the audio to be repaired; and performing audio conversion on the target spectrum graph to obtain a repaired audio of the audio to be repaired. The method can improve the audio repairing effect.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to an audio restoration method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] In financial business scenarios, it is sometimes necessary to review the audio recordings from financial systems, such as those used during loan application interviews. To facilitate audio review, it is necessary to repair the audio in financial systems to prevent poor audio quality from affecting the review results.

[0003] In traditional techniques, audio restoration in financial systems primarily relies on noise reduction. However, audio noise reduction can result in audio loss, leading to poor restoration outcomes. Summary of the Invention

[0004] Therefore, it is necessary to provide an audio restoration method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve the audio restoration effect in response to the above-mentioned technical problems.

[0005] Firstly, this application provides an audio restoration method. The method includes:

[0006] Obtain the raw spectrogram of the audio file to be repaired in the financial system;

[0007] Feature extraction is performed on the original spectrogram to obtain the original spectrogram feature map of the audio to be repaired. The original spectrogram feature map is then downsampled by different factors to obtain spectrogram feature maps of the audio to be repaired at multiple scales.

[0008] Feature extraction processing is performed on the spectral feature maps at each scale to obtain the target spectral feature maps at each scale;

[0009] Based on the audio to be repaired and the target spectral feature maps at each scale, audio reconstruction processing is performed to obtain the target spectral map of the audio to be repaired.

[0010] The target spectrogram is subjected to audio conversion to obtain the repaired audio of the audio to be repaired.

[0011] In one embodiment, the step of performing feature extraction processing on the original spectrogram to obtain the original spectrogram feature map of the audio to be repaired includes:

[0012] The original spectrogram is input into the convolutional layer of the pre-trained audio restoration model for convolution processing to obtain the feature map of the original spectrogram, which is used as the original spectrogram feature map of the audio to be restored.

[0013] The step of performing feature extraction processing on the spectral feature maps at each scale to obtain the target spectral feature maps at each scale includes:

[0014] The spectral feature maps at each scale are input into the feature extraction layers of the pre-trained audio restoration model for feature extraction processing to obtain the target spectral feature maps at each scale; each feature extraction layer includes multiple sequentially connected sub-feature extraction layers.

[0015] In one embodiment, the step of performing audio reconstruction processing based on the audio to be repaired and the target spectral feature maps at various scales to obtain the target spectral map of the audio to be repaired includes:

[0016] The target spectral feature maps at each scale are upsampled by different factors to obtain multiple target spectral feature maps at the same scale.

[0017] The audio to be repaired and the multiple target spectral feature maps of the same scale are superimposed and then input into the reconstruction layer of the pre-trained audio repair model for audio reconstruction processing to obtain the reconstructed spectral map of the audio to be repaired, which serves as the target spectral map of the audio to be repaired.

[0018] In one embodiment, the pre-trained audio restoration model is trained in the following manner:

[0019] Obtain sample audio from the financial system and the actual restored audio of the sample audio;

[0020] Obtain the original spectrogram of the sample audio;

[0021] The original spectrogram of the sample audio is input into the audio restoration model to be trained to obtain the target spectrogram of the sample audio.

[0022] The target spectrogram of the sample audio is subjected to audio transformation to obtain the predicted repair audio of the sample audio;

[0023] Based on the predicted repaired audio and the actual repaired audio of the sample audio, the audio repair model to be trained is trained to obtain the trained audio repair model, which is used as the pre-trained audio repair model.

[0024] In one embodiment, training the audio restoration model to be trained based on the predicted restored audio and the actual restored audio of the sample audio to obtain the trained audio restoration model includes:

[0025] Based on the predicted repaired audio and the actual repaired audio of the sample audio, the reconstruction loss value, the perceptual loss value, and the adversarial loss value are determined.

[0026] The reconstruction loss value, the perception loss value, and the adversarial loss value are fused together to obtain the target loss value;

[0027] The audio restoration model to be trained is trained according to the target loss value until the training termination condition is met.

[0028] The trained audio restoration model that meets the training termination condition is considered the completed audio restoration model.

[0029] In one embodiment, after performing audio conversion on the target spectrogram to obtain the repaired audio of the audio to be repaired, the method further includes:

[0030] Extract the target audio features of the repaired audio;

[0031] The target audio features and the audio to be repaired are superimposed to obtain the target repaired audio of the audio to be repaired.

[0032] In one embodiment, obtaining the original spectrogram of the audio to be repaired in the financial system includes:

[0033] The audio files to be repaired in the financial system are preprocessed to obtain preprocessed audio.

[0034] The preprocessed audio is subjected to Fourier transform to obtain a spectrogram, which serves as the original spectrogram of the audio to be repaired.

[0035] Secondly, this application also provides an audio restoration apparatus. The apparatus includes:

[0036] The acquisition module is used to acquire the original spectrogram of the audio to be repaired in the financial system;

[0037] The processing module is used to perform feature extraction processing on the original spectrogram to obtain the original spectrogram feature map of the audio to be repaired, and to perform downsampling processing on the original spectrogram feature map at different multiples to obtain the spectrogram feature maps of the audio to be repaired at multiple scales.

[0038] The extraction module is used to perform feature extraction processing on the spectral feature maps at various scales to obtain the target spectral feature maps at each scale.

[0039] The reconstruction module is used to perform audio reconstruction processing based on the audio to be repaired and the target spectral feature maps at each scale, so as to obtain the target spectral map of the audio to be repaired.

[0040] The conversion module is used to perform audio conversion on the target spectrogram to obtain the repaired audio of the audio to be repaired.

[0041] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0042] Obtain the raw spectrogram of the audio file to be repaired in the financial system;

[0043] Feature extraction is performed on the original spectrogram to obtain the original spectrogram feature map of the audio to be repaired. The original spectrogram feature map is then downsampled by different factors to obtain spectrogram feature maps of the audio to be repaired at multiple scales.

[0044] Feature extraction processing is performed on the spectral feature maps at each scale to obtain the target spectral feature maps at each scale;

[0045] Based on the audio to be repaired and the target spectral feature maps at each scale, audio reconstruction processing is performed to obtain the target spectral map of the audio to be repaired.

[0046] The target spectrogram is subjected to audio conversion to obtain the repaired audio of the audio to be repaired.

[0047] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0048] Obtain the raw spectrogram of the audio file to be repaired in the financial system;

[0049] Feature extraction is performed on the original spectrogram to obtain the original spectrogram feature map of the audio to be repaired. The original spectrogram feature map is then downsampled by different factors to obtain spectrogram feature maps of the audio to be repaired at multiple scales.

[0050] Feature extraction processing is performed on the spectral feature maps at each scale to obtain the target spectral feature maps at each scale;

[0051] Based on the audio to be repaired and the target spectral feature maps at each scale, audio reconstruction processing is performed to obtain the target spectral map of the audio to be repaired.

[0052] The target spectrogram is subjected to audio conversion to obtain the repaired audio of the audio to be repaired.

[0053] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0054] Obtain the raw spectrogram of the audio file to be repaired in the financial system;

[0055] Feature extraction is performed on the original spectrogram to obtain the original spectrogram feature map of the audio to be repaired. The original spectrogram feature map is then downsampled by different factors to obtain spectrogram feature maps of the audio to be repaired at multiple scales.

[0056] Feature extraction processing is performed on the spectral feature maps at each scale to obtain the target spectral feature maps at each scale;

[0057] Based on the audio to be repaired and the target spectral feature maps at each scale, audio reconstruction processing is performed to obtain the target spectral map of the audio to be repaired.

[0058] The target spectrogram is subjected to audio conversion to obtain the repaired audio of the audio to be repaired.

[0059] The aforementioned audio method, apparatus, computer equipment, storage medium, and computer program product acquire the original spectrogram of the audio to be repaired from the financial system, and perform feature extraction processing on the original spectrogram to obtain the original spectral feature map of the audio to be repaired. Then, the original spectral feature map is downsampled at different ratios to obtain spectral feature maps at multiple scales of the audio to be repaired. Next, feature extraction processing is performed on the spectral feature maps at each scale to obtain target spectral feature maps at each scale. Based on the audio to be repaired and the target spectral feature maps at each scale, audio reconstruction processing is performed to obtain the target spectral map of the audio to be repaired. Finally, audio conversion is performed on the target spectral map to obtain the repaired audio. In this way, during audio restoration, the audio to be restored is converted into a spectrogram for processing. The differences in feature details and the structural similarities between target spectrogram feature maps at different scales are utilized to assist in the reconstruction of the target spectrogram. This results in higher quality target spectrograms reconstructed from the audio to be restored and the target spectrogram feature maps at various scales, leading to higher audio quality in the restored audio obtained from the target spectrogram conversion. This avoids the audio loss that often occurs in audio restoration through noise reduction, resulting in poor audio restoration effects, thus improving the overall audio restoration effect. Simultaneously, feature extraction and downsampling are performed on the original spectrogram to obtain spectrogram feature maps at different scales. Further feature extraction processing on these different scale spectrogram feature maps allows for continuous optimization of the features at different scales, resulting in higher quality reconstructed target spectrograms. This, in turn, leads to better restoration effects in the audio obtained from the target spectrogram conversion, further enhancing the audio restoration outcome. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating an audio restoration method in one embodiment;

[0061] Figure 2 This is a schematic diagram of the structure of an audio restoration model in one embodiment;

[0062] Figure 3 This is a flowchart illustrating the training steps of an audio restoration model in one embodiment.

[0063] Figure 4 This is a schematic diagram illustrating the training process of an audio restoration model to be trained in one embodiment.

[0064] Figure 5 This is a flowchart illustrating the audio restoration method in another embodiment;

[0065] Figure 6 This is a structural block diagram of an audio restoration device in one embodiment;

[0066] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] It should be noted that the audio restoration method, apparatus, computer equipment, storage medium, and computer program products provided in this application can be used in the field of financial technology. For example, they can be used to reconstruct the original spectrogram of the audio to be restored in a financial system to obtain the target spectrogram of the audio to be restored, and then convert it to obtain the restored audio, thus avoiding the loss of audio information during the audio restoration process and improving the audio restoration effect. They can also be used in other related fields, such as in the field of audio processing technology, to perform a series of restoration processes on the audio to be restored to obtain restored audio with a high audio restoration effect.

[0069] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0070] In one embodiment, such as Figure 1As shown, an audio restoration method is provided. This embodiment illustrates the application of this method to a server. It is understood that this method can also be applied to a terminal, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc.; the server can be a standalone server or a server cluster composed of multiple servers. In this embodiment, the method includes the following steps:

[0071] Step S101: Obtain the original spectrogram of the audio to be repaired in the financial system.

[0072] Here, "financial system" refers to the banking system; "audio awaiting repair in the financial system" refers to low-quality audio within the financial system that requires repair, such as audio from loan interviews or bank card applications. In real-world scenarios, audio from loan interviews and bank card applications awaiting repair often contains some noise.

[0073] The original spectrogram refers to the spectrogram obtained by performing a Fourier transform on the audio to be restored, which is used to represent the relationship between frequency and amplitude. It should be noted that this application treats the original spectrogram as an image and the audio restoration as an image reconstruction.

[0074] Specifically, the server receives an audio repair request from the terminal for audio to be repaired in the financial system, and retrieves the audio to be repaired from the financial system's audio database according to the request; then, based on the spectrogram acquisition instructions, it processes the audio to be repaired to obtain the original spectrogram of the audio to be repaired. For example, the server performs a Fourier transform on the audio to be repaired to obtain the original spectrogram of the audio to be repaired.

[0075] For example, see reference. Figure 2 In the financial field, in response to a repair request for the input loan interview audio, the server performs a Fourier transform on the loan interview audio to obtain the original spectrogram of the loan interview audio.

[0076] Step S102: Perform feature extraction processing on the original spectrogram to obtain the original spectrogram feature map of the audio to be repaired. Perform downsampling processing on the original spectrogram feature map at different multiples to obtain spectrogram feature maps of the audio to be repaired at multiple scales.

[0077] The original spectral feature map refers to the feature map corresponding to the original spectrogram, specifically used to characterize the features corresponding to the original spectrogram of the audio to be repaired. In other words, the original spectral feature map is obtained by representing the features extracted from the original spectrogram through the feature map.

[0078] Here, downsampling processing refers to bicubic downsampling, and downsampling at different multiples refers to bicubic downsampling at different multiples, such as... Figure 2 The images show 2x bicubic downsampling and 3x bicubic downsampling. It should be noted that performing 1x bicubic downsampling on the original spectral feature map still yields the original spectral feature map.

[0079] The audio to be repaired includes spectral feature maps at multiple scales, specifically the original spectral feature map, the spectral feature map after 2x bicubic downsampling, and so on, up to the spectral feature map after nx bicubic downsampling, where n is a positive integer greater than or equal to 2. It should be noted that in practical scenarios, n can be 3, meaning the spectral feature maps at multiple scales are three different scales: the original spectral feature map, the spectral feature map after 2x bicubic downsampling, and the spectral feature map after 3x bicubic downsampling.

[0080] It should be noted that the original spectral feature map contains shallow audio features, and spectral feature maps at different scales also contain shallow audio features. Furthermore, the audio feature details contained in spectral feature maps at different scales differ, and there is a certain structural similarity between spectral feature maps at different scales.

[0081] Specifically, the server inputs the original spectrogram into a feature extraction model (such as a convolutional neural network model) for feature extraction processing to obtain the feature map corresponding to the original spectrogram, which serves as the original spectral feature map of the audio to be repaired; the original spectral feature map is then subjected to bicubic downsampling processing at different multiples to obtain spectral feature maps of the audio to be repaired at multiple scales.

[0082] For example, see reference. Figure 2 The server inputs the original spectrogram of the audio to be repaired into the convolutional layer for convolution processing to obtain the original spectrogram feature map of the audio to be repaired; and performs bicubic downsampling processing on the original spectrogram feature map of the audio to be repaired at different multiples to obtain the original spectrogram feature map, the spectrogram feature map after 2x bicubic downsampling processing, and the spectrogram feature map after 3x bicubic downsampling processing.

[0083] Step S103: Perform feature extraction processing on the spectral feature maps at each scale to obtain the target spectral feature maps at each scale.

[0084] Feature extraction processing refers to feature optimization based on a fully connected attention mechanism. Feature extraction is performed on the spectral feature maps at various scales primarily to further optimize the features of these maps. After further feature extraction, the spectral feature maps are further refined into the target spectral feature map.

[0085] Among them, the target spectral feature maps at various scales contain deep audio features, such as audio refinement features.

[0086] It should be noted that the feature details of target spectral feature maps at different scales are different, and there is a certain structural similarity between target spectral feature maps at different scales; by utilizing the structural consistency between target spectral feature maps at different scales, high-quality target spectral maps can be reconstructed.

[0087] Specifically, the server uses the feature extraction model (such as a model composed of multiple transformation networks) corresponding to the spectral feature maps at each scale to perform feature extraction processing on the spectral feature maps at each scale, thereby obtaining the target spectral feature maps at each scale.

[0088] For example, see reference. Figure 2 The server inputs the original spectral feature map (i.e., the spectral feature map after 1x bicubic downsampling), the spectral feature map after 2x bicubic downsampling, and the spectral feature map after 3x bicubic downsampling into three different transformation network branches (composed of multiple transformation networks) for attention-based feature extraction processing, resulting in three target spectral feature maps at different scales.

[0089] Step S104: Based on the audio to be repaired and the target spectral feature maps at various scales, perform audio reconstruction processing to obtain the target spectral map of the audio to be repaired.

[0090] The target spectrogram of the audio to be repaired refers to a high-quality spectrogram, such as... Figure 2 The reconstructed spectrogram shown is of much higher quality than the original spectrogram of the audio to be repaired.

[0091] It should be noted that when performing audio reconstruction, by utilizing the structural similarity between target spectral feature maps at various scales and simultaneously introducing the entire audio to be repaired (i.e., global information), a higher quality spectral map (i.e., target spectral map) can be reconstructed, thereby further improving the audio restoration effect.

[0092] Specifically, the server overlays the audio to be repaired and the target spectral feature maps at various scales, and then inputs them into the reconstruction module (such as the reconstruction module in the transform network) for audio reconstruction processing to obtain a high-quality spectral map, which serves as the target spectral map for the audio to be repaired.

[0093] Step S105: Perform audio conversion on the target spectrogram to obtain the repaired audio of the audio to be repaired.

[0094] Audio conversion of the target spectrogram refers to transforming the target spectrogram into its corresponding audio. In practical scenarios, audio conversion specifically refers to inverse Fourier transform.

[0095] Among them, repaired audio refers to the audio obtained after repairing the original audio, and its audio quality is higher than that of the original audio.

[0096] Specifically, the server performs an inverse Fourier transform on the target spectrogram to obtain the audio corresponding to the target spectrogram, which is then used as the repair audio for the audio to be repaired.

[0097] For example, the server uses librosa functions (audio processing functions) to convert the target spectrogram into the corresponding audio, which is then used as the repair audio for the audio to be repaired.

[0098] Furthermore, the server can also perform repair processing on the repaired audio of the audio to be repaired to obtain the target repaired audio; for example, the server obtains the original spectrogram of the repaired audio; inputs the original spectrogram of the repaired audio into a pre-trained audio repair model to obtain the target spectrogram of the repaired audio; and performs audio conversion on the target spectrogram of the repaired audio to obtain the target repaired audio of the audio to be repaired.

[0099] The audio restoration method provided in the above embodiments obtains the original spectrogram of the audio to be restored from the financial system, performs feature extraction processing on the original spectrogram to obtain the original spectral feature map of the audio to be restored, performs downsampling processing on the original spectral feature map at different multiples to obtain spectral feature maps of the audio to be restored at multiple scales, then performs feature extraction processing on the spectral feature maps at each scale to obtain the target spectral feature map at each scale, and performs audio reconstruction processing based on the audio to be restored and the target spectral feature maps at each scale to obtain the target spectral map of the audio to be restored; finally, performs audio conversion on the target spectral map to obtain the restored audio of the audio to be restored. In this way, during audio restoration, the audio to be restored is converted into a spectrogram for processing. The differences in feature details and the structural similarities between target spectrogram feature maps at different scales are utilized to assist in the reconstruction of the target spectrogram. This results in higher quality target spectrograms reconstructed from the audio to be restored and the target spectrogram feature maps at various scales, leading to higher audio quality in the restored audio obtained from the target spectrogram conversion. This avoids the audio loss that often occurs in audio restoration through noise reduction, resulting in poor audio restoration effects, thus improving the overall audio restoration effect. Simultaneously, feature extraction and downsampling are performed on the original spectrogram to obtain spectrogram feature maps at different scales. Further feature extraction processing on these different scale spectrogram feature maps allows for continuous optimization of the features at different scales, resulting in higher quality reconstructed target spectrograms. This, in turn, leads to better restoration effects in the audio obtained from the target spectrogram conversion, further enhancing the audio restoration outcome.

[0100] In one embodiment, step S102, which involves performing feature extraction processing on the original spectrogram to obtain the original spectral feature map of the audio to be repaired, specifically includes the following: inputting the original spectrogram into the convolutional layer of a pre-trained audio repair model for convolution processing to obtain the feature map of the original spectrogram, which serves as the original spectral feature map of the audio to be repaired. Then, step S103, which involves performing feature extraction processing on the spectral feature maps at various scales to obtain the target spectral feature maps at each scale, specifically includes the following: inputting the spectral feature maps at various scales into the respective feature extraction layers of the pre-trained audio repair model for feature extraction processing to obtain the target spectral feature maps at each scale; each feature extraction layer includes multiple sequentially connected sub-feature extraction layers.

[0101] The pre-trained audio restoration model is primarily used to reconstruct high-quality spectrograms. It is specifically trained by continuously training the model with sample audio samples. The pre-trained audio restoration model includes a convolutional layer, multiple feature extraction layers, and a reconstruction layer, as shown in the following structure: Figure 2 As shown.

[0102] The convolutional layers are primarily implemented using convolutional modules (conv) or convolutional neural networks, and are used to extract features. The reconstruction layers are primarily implemented using the reconstruction module in a transform network, and are used to reconstruct high-quality spectrograms.

[0103] Among them, the sub-feature extraction layer refers to Figure 2 The transformation network shown is specifically used for feature extraction. Each feature extraction layer consists of multiple sequentially connected sub-feature extraction layers, specifically used for feature extraction and optimization. For example, refer to... Figure 2 Each feature extraction layer consists of multiple sequentially connected transformation networks.

[0104] Specifically, the server inputs the original spectrogram into the convolutional layer of the pre-trained audio restoration model. The convolutional layer performs convolution processing on the original spectrogram to obtain the feature map of the original spectrogram, which serves as the original spectral feature map of the audio to be restored. Next, the server performs bicubic downsampling processing on the original spectral feature map at different multiples to obtain spectral feature maps at different scales. Finally, the server inputs the spectral feature maps at each scale into the respective feature extraction layers of the pre-trained audio restoration model. The respective feature extraction layers perform feature extraction processing on the spectral feature maps at the corresponding scales to obtain the target spectral feature maps at each scale.

[0105] For example, see reference. Figure 2The server inputs the original spectrogram of the audio to be repaired into a convolutional layer for convolution processing to obtain the original spectrogram feature map of the audio to be repaired; the original spectrogram feature map is then subjected to 2x bicubic downsampling and 3x bicubic downsampling to obtain the spectrogram feature map after 2x bicubic downsampling and the spectrogram feature map after 3x bicubic downsampling; the original spectrogram feature map, the spectrogram feature map after 2x bicubic downsampling, and the spectrogram feature map after 3x bicubic downsampling are then input into the corresponding feature extraction layer for feature extraction processing to obtain three target spectrogram feature maps at different scales.

[0106] In this embodiment, the original spectrogram is input into the convolutional layer of a pre-trained audio restoration model for convolution processing to obtain the original spectral feature map of the audio to be restored. Spectral feature maps at various scales are then input into the respective feature extraction layers of the pre-trained audio restoration model for feature extraction processing to obtain target spectral feature maps at each scale. This process of first performing feature extraction and downsampling on the original spectrogram to obtain spectral feature maps at different scales, followed by feature extraction processing on these different scales, allows for continuous optimization of the features of the spectral feature maps at different scales, resulting in a higher quality reconstructed target spectrogram. Furthermore, obtaining target spectral feature maps at different scales of the audio to be restored facilitates subsequent spectrogram reconstruction by fully utilizing the differences in feature details and the structural similarities between target spectral feature maps at different scales, thereby improving the quality of the reconstructed target spectrogram.

[0107] In one embodiment, step S104 above, which involves performing audio reconstruction processing based on the audio to be repaired and target spectral feature maps at various scales to obtain the target spectral map of the audio to be repaired, specifically includes the following: performing upsampling processing on the target spectral feature maps at various scales by different factors to obtain multiple target spectral feature maps at the same scale; superimposing the audio to be repaired and the multiple target spectral feature maps at the same scale, and then inputting them into the reconstruction layer of a pre-trained audio repair model for audio reconstruction processing to obtain the reconstructed spectral map of the audio to be repaired, which serves as the target spectral map of the audio to be repaired.

[0108] Here, upsampling refers to bicubic upsampling, and upsampling at different multiples refers to bicubic upsampling at different multiples, for example... Figure 2 The diagram shows 2x bicubic upsampling and 3x bicubic upsampling. It should be noted that performing 1x bicubic upsampling on the target spectral feature map corresponding to the original spectral feature map (i.e., the target spectral feature map at the first scale) still yields the same target spectral feature map.

[0109] Different upsampling factors correspond to different downsampling factors. For example, refer to... Figure 2 2x bicubic upsampling corresponds to 2x bicubic downsampling, and 3x bicubic upsampling corresponds to 3x bicubic downsampling.

[0110] The purpose of upsampling the target spectral feature maps at different scales is to transform them into the same scale, thereby obtaining multiple target spectral feature maps of the same scale, which facilitates subsequent spectral map reconstruction.

[0111] Specifically, the server performs bicubic upsampling on the target spectral feature maps at various scales to obtain multiple target spectral feature maps at the same scale; the audio to be repaired and the multiple target spectral feature maps at the same scale are superimposed to obtain the superimposed result; the superimposed result is input into the reconstruction layer of the pre-trained audio repair model, and the reconstruction layer uses the superimposed result to perform audio reconstruction processing to obtain the reconstructed spectral map of the audio to be repaired, which serves as the target spectral map of the audio to be repaired.

[0112] For example, see reference. Figure 2 The target spectral feature maps at different scales are the target spectral feature maps at the first, second, and third scales, respectively. The server performs a 2x bicubic upsampling process on the target spectral feature map at the second scale to transform it into the first scale; simultaneously, it performs a 3x bicubic upsampling process on the target spectral feature map at the third scale to transform it into the first scale, thus obtaining three target spectral feature maps at the same scale (i.e., the first scale). Next, the server overlays these three target spectral feature maps at the same scale with the input audio to be repaired, and then inputs the result into the reconstruction layer for audio reconstruction processing, obtaining a reconstructed spectrogram, which serves as the target spectrogram for the audio to be repaired.

[0113] In this embodiment, the target spectral feature maps at various scales are first upsampled by different factors to obtain multiple target spectral feature maps at the same scale. Then, the audio to be repaired and the multiple target spectral feature maps at the same scale are superimposed, and audio reconstruction is performed to obtain the reconstructed spectral map of the audio to be repaired, which serves as the target spectral map of the audio to be repaired. In this way, when performing audio reconstruction, the structural similarity between the target spectral feature maps at various scales is utilized, and the global information of the entire audio to be repaired is introduced, which can reconstruct a higher quality spectral map and further improve the audio repair effect.

[0114] In one embodiment, such as Figure 3As shown, the audio restoration method provided in this application also includes a training step for a pre-trained audio restoration model, specifically including the following steps:

[0115] Step S301: Obtain the sample audio from the financial system and the actual repaired audio from the sample audio.

[0116] Step S302: Obtain the original spectrogram of the sample audio.

[0117] Step S303: Input the original spectrogram of the sample audio into the audio restoration model to be trained to obtain the target spectrogram of the sample audio.

[0118] Step S304: Perform audio conversion on the target spectrogram of the sample audio to obtain the predicted repair audio of the sample audio.

[0119] Step S305: Based on the predicted repair audio and the actual repair audio of the sample audio, train the audio repair model to be trained to obtain the trained audio repair model, which serves as the pre-trained audio repair model.

[0120] Here, sample audio refers to audio with poor quality, actual repaired audio refers to normal audio with higher quality corresponding to the sample audio, and predicted repaired audio refers to the reconstructed audio of the sample audio. In real-world scenarios, sample audio refers to lossy audio, while actual repaired audio refers to normal lossless audio.

[0121] Specifically, the server collects normal, lossless audio as the actual audio to be repaired; the actual audio to be repaired is then subjected to audio attenuation processing, such as random clipping, frequency interference, and blanking, to obtain lossy audio, which is then used as sample audio. Next, the server performs a Fourier transform on the sample audio to obtain the original spectrogram of the sample audio. This original spectrogram is then input into the audio restoration model to be trained. The convolutional layers in the audio restoration model extract features from the original spectrogram to obtain the original spectral feature map of the sample audio. The original spectral feature map is then subjected to bicubic downsampling at different ratios to obtain spectral feature maps at different scales. Each feature extraction layer in the audio restoration model extracts features from the corresponding scale spectral feature maps to obtain target spectral feature maps at each scale of the sample audio. The target spectral feature maps at each scale are then upsampled at different ratios to obtain multiple target spectral feature maps of the same scale. The sample audio and the multiple target spectral feature maps of the same scale are then superimposed and input into the reconstruction layer of the audio restoration model for audio reconstruction to obtain the reconstructed spectrogram of the sample audio, which serves as the target spectrogram of the sample audio. Finally, an inverse Fourier transform is performed on the target spectrogram of the sample audio to obtain the predicted restored audio. Finally, the server calculates the target loss value based on the predicted and actual repaired audio from the sample audio, combined with the loss function. The audio repair model to be trained is then trained based on the target loss value until the target loss value obtained by the trained audio repair model is less than or equal to a preset threshold. The trained audio repair model is then used as the completed audio repair model, thus obtaining the pre-trained audio repair model.

[0122] For example, if the target loss value is greater than a preset threshold, the server adjusts the model parameters of the audio restoration model to be trained according to the target loss value and repeats the above training process, such as steps S303 to S305, to retrain the audio restoration model after the model parameters are adjusted, until the target loss value obtained from the trained audio restoration model is less than or equal to the preset threshold, then training stops. The trained audio restoration model at this time is the audio restoration model that has been trained.

[0123] In this embodiment, the audio restoration model to be trained is trained multiple times using sample audio from the financial system and the actual restored audio from the sample audio, resulting in a fully trained audio restoration model. This improves the quality of the target spectrogram output by the audio restoration model, thereby improving the audio quality of the restored audio obtained from the target spectrogram and ultimately enhancing the audio restoration effect.

[0124] In one embodiment, such as Figure 4 As shown, step S305 above, which trains the audio restoration model to be trained based on the predicted and actual restored audio samples, yields the trained audio restoration model. Specifically, it includes the following steps:

[0125] Step S401: Based on the predicted repair audio and the actual repair audio of the sample audio, determine the reconstruction loss value, the perceptual loss value, and the adversarial loss value.

[0126] Step S402: The reconstruction loss value, perception loss value and adversarial loss value are fused to obtain the target loss value.

[0127] Step S403: Train the audio restoration model to be trained according to the target loss value until the training termination condition is met.

[0128] Step S404: The trained audio restoration model that has reached the end of training conditions is taken as the audio restoration model that has completed training.

[0129] The reconstruction loss value refers to the mean squared error loss value, which is used to maintain the structural consistency between the predicted and actual restored audio. The perceptual loss value is used to enhance the details of the audio, and the adversarial loss value is used to improve the realism of the audio. It is mainly related to generative adversarial networks, while the discriminator only uses the adversarial loss. The target loss value is obtained by weighted summation of the reconstruction loss value, perceptual loss value, and adversarial loss value.

[0130] Among them, the training termination conditions are that the target loss value is less than or equal to the preset threshold, or the total number of training iterations reaches the preset number of training iterations. Specifically, the server obtains the square of the difference between the predicted and actual repaired audio for each sample audio, then sums the squares of these differences and calculates the average, which is used as the reconstruction loss value. Next, the server obtains the audio features of the predicted and actual repaired audio for each sample audio, and calculates the perceptual loss value using a perceptual loss function. Simultaneously, the server uses a generative adversarial network (GAN) and combines the predicted and actual repaired audio for each sample audio with an adversarial loss function to calculate the adversarial loss value. The reconstruction loss value, perceptual loss value, and adversarial loss value are then weighted and summed to obtain the target loss value. Next, the server adjusts the model parameters of the audio repair model to be trained based on the target loss value and repeats the above training process to retrain the audio repair model with adjusted model parameters until the training termination condition is met, such as the target loss value obtained from the trained audio repair model being less than or equal to a preset threshold. Finally, the trained audio repair model that has met the training termination condition is considered the completed audio repair model.

[0131] For example, the server first collects normal lossless audio as labels; then, it randomly performs clipping, RF interference, and blanking trajectory processing on the normal lossless audio to obtain lossy audio, which serves as input data. In the generative adversarial network (GAN), the generator uses a weighted sum of mean squared error loss, perceptual loss, and adversarial loss as the objective function. It trains the audio restoration model using training data (including input data and labels) until the accuracy of the trained audio restoration model reaches a threshold, at which point training stops, resulting in a trained audio restoration model. It should be noted that the discriminator in the GAN only uses adversarial loss.

[0132] Furthermore, this application can also use the audio restoration model to be trained as a generator, and utilize generative adversarial networks (GANs) to assist in training the audio restoration model, so as to obtain more realistic audio data through the audio restoration model. The GAN includes a generator and a discriminator. The generator consists of a transformation network, meaning the trained generator is an audio restoration model based on the transformation network. The discriminator consists of several convolutional layers, a fully connected layer, and a densely connected layer. The generator primarily processes the audio to be restored by first extracting features through convolutional layers, and then optimizing the extracted features through stacked transformation network base blocks. Compared to residual modules, transformation networks are better able to handle global dependencies and perform better feature optimization. Then, the original audio to be restored is superimposed through global skip connections, allowing the entire model to focus more on processing detailed information rather than structural information. Finally, the features are transformed into high-quality audio through a reconstruction module. The discriminator primarily converts the original input into numerical values; the closer to 1, the more realistic the input, and the closer to 0, the more fake. High-quality audio recovered from iterative input and high-definition audio representing the ground truth of the labels are used to convert generated samples and actual samples into numerical values ​​within the network. These values ​​are then substituted into the adversarial network training formula and used as the objective function for iterative training. The specific training process is as follows: the goal is for the discriminator to effectively distinguish between real and fake samples, while the generator can deceive this good discriminator, generating audio that is indistinguishable from genuine samples. The first step is to have the discriminator classify real samples as 1. The second step is to train the discriminator with fake samples generated under a fixed generator, classifying them as 0. After training to a certain extent, the discriminator's loss becomes small. The third step is to fix the discriminator and train the generator.

[0133] In this embodiment, based on the predicted and actual repaired audio samples, reconstruction loss, perceptual loss, and adversarial loss are determined. These losses are then fused to obtain a target loss. Finally, the audio repair model is trained using the target loss to obtain a fully trained audio repair model. By using these three loss values ​​(reconstruction loss, perceptual loss, and adversarial loss) multiple times during model training, the target spectrogram obtained from the trained audio repair model is ensured to be more natural, resulting in a more natural reconstructed audio and further improving the audio repair effect.

[0134] In one embodiment, after performing audio conversion on the target spectrogram to obtain the repaired audio of the audio to be repaired, step S105 above further includes the following: extracting the target audio features of the repaired audio; and superimposing the target audio features and the audio to be repaired to obtain the target repaired audio of the audio to be repaired.

[0135] The target audio features for audio restoration refer to the deep audio features of the restored audio. The audio quality of the target restored audio is higher than that of the restored audio.

[0136] Specifically, the server performs convolution and residual processing on the repaired audio to further optimize the audio features of the repaired audio, obtaining deep audio features of the repaired audio, which are used as the target audio features of the repaired audio; the target audio features of the repaired audio are superimposed with the audio to be repaired to obtain the target repaired audio of the audio to be repaired.

[0137] For example, the server inputs the audio to be repaired into a convolutional layer, a residual layer, and a reconstruction layer for feature extraction to obtain the target audio features of the repaired audio. These features are then superimposed on the audio to be repaired to obtain the reconstructed audio, which serves as the target audio for repair.

[0138] In this embodiment, the target audio features of the repaired audio are first extracted, and then the target audio features of the repaired audio are superimposed with the audio to be repaired to obtain the target repaired audio of the audio to be repaired. In this way, after obtaining the repaired audio of the audio to be repaired, the target audio features of the repaired audio are extracted again and superimposed with the audio to be repaired, which is beneficial for performing repair processing on the repaired audio again and further improves the audio repair effect.

[0139] In one embodiment, step S101 above, obtaining the original spectrogram of the audio to be repaired in the financial system, specifically includes the following: preprocessing the audio to be repaired in the financial system to obtain preprocessed audio; performing Fourier transform on the preprocessed audio to obtain a spectrogram, which serves as the original spectrogram of the audio to be repaired.

[0140] Preprocessing refers to removing audio interference elements such as abnormal frequencies and pulse spikes.

[0141] Specifically, the server obtains the audio to be repaired from the financial system and preprocesses it, such as removing abnormal frequencies, spikes, and other audio interference elements, to obtain preprocessed audio. Then, the preprocessed audio is subjected to Fourier transform to obtain the corresponding spectrogram, which serves as the original spectrogram of the audio to be repaired.

[0142] In this embodiment, the audio to be repaired is first preprocessed, and then the preprocessed audio is subjected to Fourier transform to obtain the original spectrogram of the audio to be repaired. In this way, the audio to be repaired is preprocessed before obtaining the original spectrogram, which can perform preliminary repair on the audio to be repaired, ensuring that the original spectrogram obtained later is more accurate, thereby improving the quality of the target spectrogram reconstructed from the original spectrogram and further improving the audio repair effect.

[0143] In one embodiment, such as Figure 5 As shown, another audio repair method is provided. Taking the application of this method to a server as an example, the steps include:

[0144] Step S501: Preprocess the audio to be repaired in the financial system to obtain preprocessed audio; perform Fourier transform on the preprocessed audio to obtain a spectrogram, which is used as the original spectrogram of the audio to be repaired.

[0145] Step S502: Input the original spectrogram into the convolutional layer of the pre-trained audio restoration model for convolution processing to obtain the feature map of the original spectrogram, which serves as the original spectrogram feature map of the audio to be restored.

[0146] Step S503: The original spectral feature map is downsampled by different factors to obtain spectral feature maps of multiple scales of the audio to be repaired.

[0147] Step S504: Input the spectral feature maps at each scale into the feature extraction layers of the pre-trained audio restoration model for feature extraction processing to obtain the target spectral feature maps at each scale; each feature extraction layer includes multiple sequentially connected sub-feature extraction layers.

[0148] Step S505: Upsample the target spectral feature maps at different scales by different multiples to obtain multiple target spectral feature maps at the same scale.

[0149] Step S506: After superimposing the audio to be repaired and multiple target spectral feature maps of the same scale, the data is input into the reconstruction layer of the pre-trained audio repair model for audio reconstruction processing to obtain the reconstructed spectral map of the audio to be repaired, which serves as the target spectral map of the audio to be repaired.

[0150] Step S507: Perform audio conversion on the target spectrogram to obtain the repaired audio of the audio to be repaired.

[0151] Step S508: Extract the target audio features of the audio to be repaired; superimpose the target audio features and the audio to be repaired to obtain the target repaired audio of the audio to be repaired.

[0152] The audio restoration method provided in the above embodiments converts the audio to be restored into a spectrogram for processing. It utilizes the differences in feature details between target spectrogram feature maps at different scales of the audio to be restored, and the structural similarity between these target spectrogram feature maps at different scales, to assist in the reconstruction of the target spectrogram. This results in higher quality target spectrograms reconstructed based on the audio to be restored and the target spectrogram feature maps at various scales, thereby improving the audio quality of the restored audio obtained from the target spectrogram conversion. This avoids the audio loss that often occurs in audio restoration through noise reduction, leading to poor audio restoration results, thus improving the audio restoration effect. Furthermore, by first performing feature extraction and downsampling processing on the original spectrogram to obtain spectrogram feature maps at different scales, and then performing feature extraction processing on these spectrogram feature maps at different scales, the features of the spectrogram feature maps at different scales can be continuously optimized, resulting in higher quality reconstructed target spectrograms. This, in turn, leads to better restoration results for the restored audio obtained from the target spectrogram conversion, further enhancing the audio restoration effect.

[0153] In one embodiment, to more clearly illustrate the audio restoration method provided by the embodiments of this application, the following detailed description of the audio restoration method is based on a specific embodiment. In one embodiment, refer to... Figure 2 This application also provides an audio restoration method based on transform networks. It uses a transform network module as the generator to restore audio data, and then uses a generative adversarial network (GAN) to perform adversarial learning, combining reconstruction loss, perceptual loss, and adversarial loss to obtain a trained generator, i.e., an audio restoration model. Finally, the trained audio restoration model is used to restore the audio to be restored, resulting in a more natural restored audio. The specific details are as follows:

[0154] The server converts the audio into a spectrogram for processing. It uses bicubic downsampling to obtain spectrograms at different scales and leverages the structural consistency of these spectrograms to repair damaged audio. An audio restoration model is built using a transform network module and trained with a generative adversarial network. The trained model then generates more realistic audio data. (Reference) Figure 2 The server first performs a Fourier transform on the input audio to be repaired to obtain the original spectrogram of the audio to be repaired; then it inputs the original spectrogram of the audio to be repaired into a pre-trained audio repair model for a series of processing to obtain the reconstructed spectrogram of the audio to be repaired; finally, it performs an inverse Fourier transform on the reconstructed spectrogram of the audio to be repaired to obtain the repaired audio.

[0155] The audio restoration method based on transform networks provided in the above embodiments utilizes generative adversarial networks (GANs) for adversarial learning during network training. By comprehensively considering reconstruction loss, perceptual loss, and adversarial loss, the trained audio restoration model can generate more natural audio restoration effects, thereby ensuring high-quality restored audio and improving the overall audio restoration effect. Furthermore, during audio restoration, the audio to be restored is converted into a spectrogram for processing, and the target spectral feature maps at different scales are comprehensively considered. This results in a higher-quality reconstructed target spectrogram, further enhancing the audio restoration effect.

[0156] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0157] Based on the same inventive concept, this application also provides an audio restoration apparatus for implementing the audio restoration method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more audio restoration apparatus embodiments provided below can be found in the limitations of the audio restoration method described above, and will not be repeated here.

[0158] In one embodiment, such as Figure 6As shown, an audio restoration device is provided, comprising: an acquisition module 610, a processing module 620, an extraction module 630, a reconstruction module 640, and a conversion module 650, wherein:

[0159] The acquisition module 610 is used to acquire the original spectrogram of the audio to be repaired in the financial system.

[0160] The processing module 620 is used to perform feature extraction processing on the original spectrogram to obtain the original spectrogram feature map of the audio to be repaired, and to perform downsampling processing on the original spectrogram feature map at different multiples to obtain spectrogram feature maps of the audio to be repaired at multiple scales.

[0161] The extraction module 630 is used to perform feature extraction processing on the spectral feature maps at various scales to obtain the target spectral feature maps at various scales.

[0162] The reconstruction module 640 is used to perform audio reconstruction processing based on the audio to be repaired and the target spectral feature maps at various scales to obtain the target spectral map of the audio to be repaired.

[0163] The conversion module 650 is used to perform audio conversion on the target spectrogram to obtain the repaired audio of the audio to be repaired.

[0164] In one embodiment, the processing module 620 is further configured to input the original spectrogram into the convolutional layer of the pre-trained audio restoration model for convolution processing to obtain the feature map of the original spectrogram, which serves as the original spectrogram feature map of the audio to be restored.

[0165] The extraction module 630 is also used to input the spectral feature maps at various scales into the feature extraction layers in the pre-trained audio restoration model for feature extraction processing to obtain the target spectral feature maps at various scales; each feature extraction layer includes multiple sequentially connected sub-feature extraction layers.

[0166] In one embodiment, the reconstruction module 640 is further configured to perform upsampling processing on the target spectral feature maps at various scales by different multiples to obtain multiple target spectral feature maps at the same scale; after superimposing the audio to be repaired and the multiple target spectral feature maps at the same scale, it is input into the reconstruction layer in the pre-trained audio repair model for audio reconstruction processing to obtain the reconstructed spectral map of the audio to be repaired, which serves as the target spectral map of the audio to be repaired.

[0167] In one embodiment, the audio restoration apparatus provided in this application further includes a model training module, used to acquire sample audio from a financial system and actual restored audio of the sample audio; acquire the original spectrogram of the sample audio; input the original spectrogram of the sample audio into the audio restoration model to be trained to obtain the target spectrogram of the sample audio; perform audio conversion on the target spectrogram of the sample audio to obtain the predicted restored audio of the sample audio; and train the audio restoration model to be trained based on the predicted restored audio and the actual restored audio of the sample audio to obtain the trained audio restoration model, which serves as the pre-trained audio restoration model.

[0168] In one embodiment, the model training module is further configured to determine the reconstruction loss value, perceptual loss value, and adversarial loss value based on the predicted repair audio and the actual repair audio of the sample audio; fuse the reconstruction loss value, perceptual loss value, and adversarial loss value to obtain the target loss value; train the audio repair model to be trained based on the target loss value until the training termination condition is met; and use the trained audio repair model that has met the training termination condition as the trained audio repair model.

[0169] In one embodiment, the audio repair apparatus provided in this application further includes a repair module for extracting target audio features of the audio to be repaired; and superimposing the target audio features and the audio to be repaired to obtain the target repaired audio of the audio to be repaired.

[0170] In one embodiment, the acquisition module 610 is further configured to preprocess the audio to be repaired in the financial system to obtain preprocessed audio; and to perform Fourier transform on the preprocessed audio to obtain a spectrogram, which serves as the original spectrogram of the audio to be repaired.

[0171] Each module in the aforementioned audio restoration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0172] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data such as spectrograms, spectral characteristic maps, and audio restoration data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an audio restoration method.

[0173] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0174] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0175] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0176] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0177] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0178] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0179] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An audio restoration method, characterized in that, The method includes: Obtain the raw spectrogram of the audio file to be repaired in the financial system; The original spectrogram is input into the convolutional layer of the pre-trained audio restoration model for convolution processing to obtain the original spectrogram feature map of the audio to be restored. The original spectrogram feature map is then downsampled by different factors to obtain spectrogram feature maps of the audio to be restored at multiple scales. The spectral feature maps at each scale are input into the feature extraction layers of the pre-trained audio restoration model for feature extraction processing to obtain the target spectral feature maps at each scale. Each feature extraction layer includes multiple sequentially connected sub-feature extraction layers. The target spectral feature maps at each scale are upsampled by different factors to obtain multiple target spectral feature maps at the same scale. The audio to be repaired and the multiple target spectral feature maps of the same scale are superimposed and then input into the reconstruction layer of the pre-trained audio repair model for audio reconstruction processing to obtain the target spectral map of the audio to be repaired. The target spectrogram is subjected to audio conversion to obtain the repaired audio of the audio to be repaired.

2. The method according to claim 1, characterized in that, The pre-trained audio restoration model was trained in the following manner: Obtain sample audio from the financial system and the actual restored audio of the sample audio; Obtain the original spectrogram of the sample audio; The original spectrogram of the sample audio is input into the audio restoration model to be trained to obtain the target spectrogram of the sample audio. The target spectrogram of the sample audio is subjected to audio transformation to obtain the predicted repair audio of the sample audio; Based on the predicted repaired audio and the actual repaired audio of the sample audio, the audio repair model to be trained is trained to obtain the trained audio repair model, which is used as the pre-trained audio repair model.

3. The method according to claim 2, characterized in that, The step of training the audio restoration model to be trained based on the predicted restored audio and the actual restored audio from the sample audio, to obtain the trained audio restoration model, includes: Based on the predicted repaired audio and the actual repaired audio of the sample audio, the reconstruction loss value, the perceptual loss value, and the adversarial loss value are determined. The reconstruction loss value, the perception loss value, and the adversarial loss value are fused together to obtain the target loss value; The audio restoration model to be trained is trained according to the target loss value until the training termination condition is met. The trained audio restoration model that meets the training termination condition is considered the completed audio restoration model.

4. The method according to claim 1, characterized in that, After performing audio conversion on the target spectrogram to obtain the repaired audio of the audio to be repaired, the method further includes: Extract the target audio features of the repaired audio; The target audio features and the audio to be repaired are superimposed to obtain the target repaired audio of the audio to be repaired.

5. The method according to claim 1, characterized in that, The acquisition of the original spectrogram of the audio to be repaired from the financial system includes: The audio files to be repaired in the financial system are preprocessed to obtain preprocessed audio. The preprocessed audio is subjected to Fourier transform to obtain a spectrogram, which serves as the original spectrogram of the audio to be repaired.

6. An audio restoration device, characterized in that, The device includes: The acquisition module is used to acquire the original spectrogram of the audio to be repaired in the financial system; The processing module is used to perform convolution processing on the original spectrogram input into the convolutional layer of the pre-trained audio restoration model to obtain the original spectrogram feature map of the audio to be restored, and to perform downsampling processing on the original spectrogram feature map at different multiples to obtain spectrogram feature maps of the audio to be restored at multiple scales. The extraction module is used to input the spectral feature maps at various scales into the feature extraction layers of the pre-trained audio restoration model for feature extraction processing to obtain the target spectral feature maps at each scale. Each feature extraction layer includes multiple sequentially connected sub-feature extraction layers. The reconstruction module is used to upsample the target spectral feature maps at various scales by different factors to obtain multiple target spectral feature maps at the same scale; after superimposing the audio to be repaired and the multiple target spectral feature maps at the same scale, the data is input into the reconstruction layer of the pre-trained audio restoration model for audio reconstruction processing to obtain the target spectral map of the audio to be repaired. The conversion module is used to perform audio conversion on the target spectrogram to obtain the repaired audio of the audio to be repaired.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images