Audio enhancement method and apparatus, computer device and storage medium

Abstract

The application relates to an audio enhancement method and device, computer equipment, a storage medium and a computer program product, which can be used in the technical field of audio processing and can also be used in the field of financial technology or other related fields. The method comprises the following steps: performing segmentation processing on to-be-processed audio of a financial system to obtain a plurality of audio segments of the to-be-processed audio; acquiring alignment features between each audio segment and a reference audio segment of each audio segment; performing fusion processing on initial audio features of each audio segment and the alignment features to obtain fusion features of each audio segment; performing feature extraction processing on the fusion features of each audio segment to obtain target audio features of each audio segment; adding the target audio features of each audio segment to each audio segment to obtain an enhanced audio segment of each audio segment; and combining the enhanced audio segments of each audio segment to obtain an enhanced audio. The method can improve the audio enhancement effect.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to an audio enhancement method, apparatus, computer device, storage medium, and computer program product. Background Technology

[0002] In financial business scenarios, it is sometimes necessary to review the audio recordings from financial systems, such as those used during loan interviews. To facilitate audio review, audio enhancement processing is required to prevent poor audio quality from affecting the review results.

[0003] In traditional techniques, audio enhancement in financial systems primarily relies on filtering. However, simply using filtering can easily miss crucial audio information, resulting in poor enhancement quality. Summary of the Invention

[0004] Therefore, it is necessary to provide an audio enhancement method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve the audio enhancement effect in response to the above-mentioned technical problems.

[0005] Firstly, this application provides an audio enhancement method. The method includes:

[0006] The audio to be processed in the financial system is segmented to obtain multiple audio segments of the audio to be processed;

[0007] The alignment features between each audio segment and its reference audio segment are obtained, and the initial audio features and alignment features of each audio segment are fused to obtain the fused features of each audio segment.

[0008] The fusion features of each audio segment are processed by feature extraction to obtain the target audio features of each audio segment;

[0009] The target audio features of each audio segment are added to each audio segment to obtain the enhanced audio segment of each audio segment;

[0010] The enhanced audio segments of each audio segment are combined to obtain the enhanced audio of the audio to be processed.

[0011] In one embodiment, obtaining the alignment features between each audio segment and a reference audio segment, and fusing the initial audio features and alignment features of each audio segment to obtain the fused features of each audio segment, includes:

[0012] Each audio segment and its reference audio segment are input into a deformable convolutional layer in a pre-trained audio enhancement model for deformable convolution processing to obtain alignment features between each audio segment and its reference audio segment.

[0013] The initial audio features and alignment features of each audio segment are spliced ​​together to obtain the splicing features of each audio segment, which serve as the fusion features of each audio segment.

[0014] In one embodiment, the pre-trained audio enhancement model further includes a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer connected in sequence. The first feature extraction layer includes a first convolutional layer. The second feature extraction layer includes a second convolutional layer, a first dense block, a second dense block, a third dense block, a splicing layer, a third convolutional layer, and a fourth convolutional layer connected in sequence. The first dense block and the second dense block are also connected to the splicing layer. The third feature extraction layer includes a fifth convolutional layer, a residual layer, a sixth convolutional layer, and a reconstruction layer connected in sequence.

[0015] The step of performing feature extraction processing on the fusion features of each audio segment to obtain the target audio features of each audio segment includes:

[0016] The fused features of each audio segment are input into the first feature extraction layer for convolution processing to obtain the first audio features of each audio segment;

[0017] The first audio feature of each audio segment is input into the second feature extraction layer for multiple feature extraction processes to obtain the second audio feature of each audio segment.

[0018] The first and second audio features of each audio segment are superimposed and then input into the third feature extraction layer for multiple feature extraction processes to obtain the target audio features of each audio segment.

[0019] In one embodiment, the pre-trained audio enhancement model is trained in the following manner:

[0020] Acquire sample audio from the financial system and the actual enhanced audio of the sample audio;

[0021] The sample audio is segmented to obtain multiple sample audio segments;

[0022] Each sample audio segment and its reference sample audio segment are input into the audio enhancement model to be trained to obtain the predicted enhanced audio segment for each sample audio segment.

[0023] The predicted enhanced audio segments of each sample audio segment are combined to obtain the predicted enhanced audio of the sample audio.

[0024] Based on the predicted enhanced audio and the actual enhanced audio of the sample audio, the audio enhancement model to be trained is trained to obtain the trained audio enhancement model, which is used as the pre-trained audio enhancement model.

[0025] In one embodiment, training the audio enhancement model to be trained based on the predicted enhanced audio and the actual enhanced audio of the sample audio to obtain the trained audio enhancement model includes:

[0026] Based on the predicted enhanced audio and the actual enhanced audio of the sample audio, determine the reconstruction loss value and the perceptual loss value;

[0027] The reconstruction loss value and the perception loss value are fused together to obtain the target loss value;

[0028] The audio enhancement model to be trained is trained according to the target loss value and the preset channel pruning method until the training termination condition is met.

[0029] The trained audio enhancement model that meets the training termination condition is considered the completed audio enhancement model.

[0030] In one embodiment, combining the enhanced audio segments of each audio segment to obtain the enhanced audio of the audio to be processed includes:

[0031] The enhanced audio segments of each audio segment are input into a pre-trained audio quality recognition model to obtain the audio quality prediction results of the enhanced audio segments of each audio segment;

[0032] If the audio quality prediction result of the enhanced audio segment of each audio segment meets the preset audio quality conditions, the order of arrangement of each audio segment is determined;

[0033] According to the stated arrangement, the enhanced audio segments of each audio segment are combined to obtain the enhanced audio of the audio to be processed.

[0034] In one embodiment, after combining the enhanced audio segments of each audio segment to obtain the enhanced audio of the audio to be processed, the method further includes:

[0035] Extract the target audio features of the enhanced audio;

[0036] The target audio features of the enhanced audio are added to the audio to be processed to obtain the target enhanced audio of the audio to be processed.

[0037] Secondly, this application also provides an audio enhancement device. The device includes:

[0038] The audio segmentation module is used to segment the audio to be processed in the financial system to obtain multiple audio segments of the audio to be processed.

[0039] The feature fusion module is used to obtain the alignment features between each audio segment and the reference audio segment of each audio segment, and to fuse the initial audio features and alignment features of each audio segment to obtain the fused features of each audio segment.

[0040] The feature extraction module is used to perform feature extraction processing on the fusion features of each audio segment to obtain the target audio features of each audio segment;

[0041] The feature addition module is used to add the target audio features of each audio segment to each audio segment to obtain the enhanced audio segment of each audio segment;

[0042] An audio combination module is used to combine the enhanced audio segments of each audio segment to obtain the enhanced audio of the audio to be processed.

[0043] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0044] The audio to be processed in the financial system is segmented to obtain multiple audio segments of the audio to be processed;

[0045] The alignment features between each audio segment and its reference audio segment are obtained, and the initial audio features and alignment features of each audio segment are fused to obtain the fused features of each audio segment.

[0046] The fusion features of each audio segment are processed by feature extraction to obtain the target audio features of each audio segment;

[0047] The target audio features of each audio segment are added to each audio segment to obtain the enhanced audio segment of each audio segment;

[0048] The enhanced audio segments of each audio segment are combined to obtain the enhanced audio of the audio to be processed.

[0049] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0050] The audio to be processed in the financial system is segmented to obtain multiple audio segments of the audio to be processed;

[0051] The alignment features between each audio segment and its reference audio segment are obtained, and the initial audio features and alignment features of each audio segment are fused to obtain the fused features of each audio segment.

[0052] The fusion features of each audio segment are processed by feature extraction to obtain the target audio features of each audio segment;

[0053] The target audio features of each audio segment are added to each audio segment to obtain the enhanced audio segment of each audio segment;

[0054] The enhanced audio segments of each audio segment are combined to obtain the enhanced audio of the audio to be processed.

[0055] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0056] The audio to be processed in the financial system is segmented to obtain multiple audio segments of the audio to be processed;

[0057] The alignment features between each audio segment and its reference audio segment are obtained, and the initial audio features and alignment features of each audio segment are fused to obtain the fused features of each audio segment.

[0058] The fusion features of each audio segment are processed by feature extraction to obtain the target audio features of each audio segment;

[0059] The target audio features of each audio segment are added to each audio segment to obtain the enhanced audio segment of each audio segment;

[0060] The enhanced audio segments of each audio segment are combined to obtain the enhanced audio of the audio to be processed.

[0061] The aforementioned audio enhancement method, apparatus, computer equipment, storage medium, and computer program product segment the audio to be processed in a financial system to obtain multiple audio segments; then, they acquire the alignment features between each audio segment and its reference audio segment, fuse the initial audio features and alignment features of each audio segment to obtain the fused features of each audio segment; next, they perform feature extraction processing on the fused features of each audio segment to obtain the target audio features of each audio segment; finally, they add the target audio features of each audio segment to each audio segment to obtain the enhanced audio segment of each audio segment, and combine the enhanced audio segments of each audio segment to obtain the enhanced audio of the audio to be processed. In this way, during audio enhancement, the audio to be processed is first divided into multiple audio segments. The alignment features between each audio segment and its reference segment are used to help obtain the target audio features of each segment. These target audio features are then used to enhance each segment, resulting in an enhanced audio segment. Finally, these enhanced audio segments are combined to obtain the enhanced audio of the audio to be processed. This approach, by enhancing audio segments separately and then combining the enhanced segments, fully considers the information of each segment, avoiding the omission of key audio information and thus improving the audio enhancement effect. Furthermore, when extracting features from each audio segment, the alignment features between each segment and its reference segment are comprehensively considered, which helps to further optimize the audio features of each segment. This makes the reconstructed enhanced audio segments more accurate, and consequently, the enhanced audio obtained by combining these segments is more accurate, further improving the audio enhancement effect. This approach also avoids the drawback of using filtering methods to enhance audio, which can easily miss key audio information and result in poor audio enhancement. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating an audio enhancement method in one embodiment;

[0063] Figure 2 This is a schematic diagram of the structure of an audio enhancement model in one embodiment;

[0064] Figure 3 This is a schematic diagram of the deformable convolutional layer in one embodiment;

[0065] Figure 4 This is a flowchart illustrating the steps for extracting target audio features for each audio segment in one embodiment;

[0066] Figure 5 This is a flowchart illustrating the training steps of an audio enhancement model in one embodiment;

[0067] Figure 6 This is a flowchart illustrating the audio enhancement method in another embodiment;

[0068] Figure 7 This is a structural block diagram of an audio enhancement device in one embodiment;

[0069] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0071] It should be noted that the audio enhancement method, apparatus, computer equipment, storage medium and computer program products provided in this application can be used in the field of financial technology, such as enhancing the audio to be processed in a financial system by dividing the audio into audio segments to obtain enhanced audio of the audio to be processed, avoiding the omission of key audio information, thereby improving the audio enhancement effect; it can also be used in other related fields, such as in the field of audio processing technology, performing a series of audio enhancement processes on the audio to be processed to obtain enhanced audio with a higher audio enhancement effect.

[0072] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0073] In one embodiment, such as Figure 1 As shown, an audio enhancement method is provided. This embodiment illustrates the application of this method to a server. It is understood that this method can also be applied to a terminal, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc.; the server can be a standalone server or a server cluster composed of multiple servers. In this embodiment, the method includes the following steps:

[0074] Step S101: The audio to be processed in the financial system is segmented to obtain multiple audio segments.

[0075] The financial system refers to the banking system; the audio to be processed in the financial system refers to low-quality audio in the financial system that needs to be enhanced, such as audio during loan interviews or bank card applications.

[0076] The segmentation process refers to dividing the audio to be processed into multiple audio segments. For example, a 10-second audio segment can be divided into 10 segments, each segment lasting 1 second. It should be noted that the duration of each audio segment can be determined based on actual circumstances, and this application does not impose any specific limitations. Furthermore, the durations of different audio segments can be the same or different.

[0077] It should be noted that the multiple audio segments to be processed are arranged in chronological order.

[0078] Specifically, the server receives an audio enhancement request from the terminal for audio to be processed in the financial system. Based on the request, it retrieves the audio to be processed from the audio database. Then, according to the audio segmentation instructions, it segments the audio to be processed, obtaining multiple audio segments. For example, the server inputs the audio to be processed into a pre-trained lossless audio segmentation model for lossless segmentation, obtaining multiple audio segments. The pre-trained lossless audio segmentation model is primarily obtained by training a neural network model multiple times using sample audio data.

[0079] For example, in the financial field, the server segments the audio of the loan interview to be processed, resulting in multiple audio segments, namely audio segment 1, audio segment 2, audio segment 3, ... audio segment n, where n is a positive integer greater than or equal to 2.

[0080] Furthermore, the server can first perform audio preprocessing on the audio to be processed, such as removing noise, abnormal frequencies, and audio interference elements such as pulse spikes, to obtain preprocessed audio; then the preprocessed audio is input into a pre-trained lossless audio segmentation model for lossless segmentation processing to obtain multiple audio segments of the preprocessed audio, which are used as multiple audio segments of the audio to be processed.

[0081] Step S102: Obtain the alignment features between each audio segment and its reference audio segment, and fuse the initial audio features and alignment features of each audio segment to obtain the fused features of each audio segment.

[0082] In this context, the reference audio segment for each audio segment refers to the adjacent audio segment of each audio segment; for example, the reference audio segment for the first audio segment is the next audio segment after the first audio segment, i.e., the second audio segment; the reference audio segment for intermediate audio segments other than the first and last audio segments refers to the previous and next audio segments of the intermediate audio segment, such as the reference audio segment for the second audio segment being the first and third audio segments; the reference audio segment for the last audio segment is the previous audio segment of the last audio segment, i.e., the second to last audio segment.

[0083] Among them, alignment features are used to represent the audio relationship between an audio segment and a reference audio segment, specifically referring to multi-scale alignment features.

[0084] The initial audio features of an audio segment refer to the shallow audio features of the audio segment, which are relatively close to the input audio segment; the target audio features of an audio segment refer to the deep audio features of the audio segment, specifically the refined features of the audio segment, which are relatively close to the corresponding enhanced audio segment output.

[0085] Among them, fusion processing refers to concatenating the initial audio features and alignment features of an audio segment together to obtain a concatenated feature with a higher feature dimension; fusion feature refers to the concatenated feature obtained after concatenating the initial audio features and alignment features of an audio segment together.

[0086] Specifically, the server identifies a reference audio segment for each audio segment from multiple audio segments of the audio to be processed. It then performs deformable convolution processing on each audio segment and its reference audio segment to obtain alignment features between them. For example, the server uses a deformable convolutional network to extract alignment features between each audio segment and its reference audio segment. Next, the server performs feature extraction processing on each audio segment to obtain initial audio features. Finally, it concatenates the initial audio features and alignment features of each audio segment to obtain concatenated features, which serve as the fusion features for each audio segment.

[0087] For example, see reference. Figure 2In the financial field, when a server processes an audio segment, it needs to utilize the current audio segment and the two audio segments preceding and following it. This involves dividing the current audio segment and the two preceding / following segments into two groups: (current audio segment, previous audio segment) and (current audio segment, next audio segment). Then, these groups are input into corresponding deformable convolutional layers. The deformable convolutional layers extract alignment features from these groups, resulting in two alignment features: one between the current audio segment and the previous audio segment, and the other between the current audio segment and the next audio segment. Next, the initial audio features of the current audio segment are extracted, and these three alignment features are input into a concatenation layer for concatenation, resulting in a concatenated feature that serves as the fusion feature of the current audio segment.

[0088] Step S103: Perform feature extraction processing on the fusion features of each audio segment to obtain the target audio features of each audio segment.

[0089] Among them, target audio features refer to the deep audio features obtained after multiple feature extraction processes on the fusion features of audio segments.

[0090] Specifically, the server performs a series of feature extraction processes on the fusion features of each audio segment to continuously refine the features of each audio segment, thereby obtaining the deep audio features of each audio segment as the target audio features of each audio segment.

[0091] For example, see reference. Figure 2 The server utilizes Figure 2 The network within the dashed box performs a series of feature extraction processes on the splicing features of the current audio segment to continuously optimize the splicing features of the current audio segment, thereby obtaining the target audio features of the current audio segment.

[0092] Step S104: Add the target audio features of each audio segment to each audio segment to obtain the enhanced audio segment of each audio segment.

[0093] Among them, enhanced audio segments refer to high-quality audio segments corresponding to audio segments, such as... Figure 2 The reconstructed audio clip shown.

[0094] Adding the target audio features of each audio segment to each audio segment means superimposing the target audio features of each audio segment onto each audio segment.

[0095] Specifically, the server overlays the target audio features of each audio segment with each audio segment to obtain the reconstructed audio segment of each audio segment, which corresponds to the enhanced audio segment of each audio segment.

[0096] For example, see reference. Figure 2 After obtaining the target audio features of the current audio segment, the server overlays the target audio features of the current audio segment with the current audio segment to obtain the reconstructed audio segment of the current audio segment, which serves as the enhanced audio segment of the current audio segment.

[0097] Step S105: Combine the enhanced audio segments of each audio segment to obtain the enhanced audio of the audio to be processed.

[0098] Among them, the enhanced audio of the audio to be processed refers to the high-quality audio of the audio to be processed, such as... Figure 2 The reconstructed audio consists of the various reconstructed audio segments shown.

[0099] Specifically, the server obtains the arrangement order of each audio segment, and combines the enhanced audio segments of each audio segment according to the arrangement order to obtain the enhanced audio of the audio to be processed.

[0100] For example, the server combines the reconstructed audio segments of audio segment 1, audio segment 2, audio segment 3, ... audio segment n of the loan interview audio to obtain the reconstructed audio of the loan interview audio.

[0101] The audio enhancement method provided in the above embodiments involves segmenting the audio to be processed in a financial system to obtain multiple audio segments; then, obtaining the alignment features between each audio segment and its reference audio segment; fusing the initial audio features and alignment features of each audio segment to obtain the fused features of each audio segment; next, performing feature extraction on the fused features of each audio segment to obtain the target audio features of each audio segment; finally, adding the target audio features of each audio segment to each audio segment to obtain the enhanced audio segments of each audio segment; and combining the enhanced audio segments of each audio segment to obtain the enhanced audio of the audio to be processed. In this way, during audio enhancement, the audio to be processed is first divided into multiple audio segments. The alignment features between each audio segment and its reference segment are used to help obtain the target audio features of each segment. These target audio features are then used to enhance each segment, resulting in an enhanced audio segment. Finally, these enhanced audio segments are combined to obtain the enhanced audio of the audio to be processed. This approach, by enhancing audio segments separately and then combining the enhanced segments, fully considers the information of each segment, avoiding the omission of key audio information and thus improving the audio enhancement effect. Furthermore, when extracting features from each audio segment, the alignment features between each segment and its reference segment are comprehensively considered, which helps to further optimize the audio features of each segment. This makes the reconstructed enhanced audio segments more accurate, and consequently, the enhanced audio obtained by combining these segments is more accurate, further improving the audio enhancement effect. This approach also avoids the drawback of using filtering methods to enhance audio, which can easily miss key audio information and result in poor audio enhancement.

[0102] In one embodiment, step S102 above, obtaining the alignment features between each audio segment and its reference audio segment, and fusing the initial audio features and alignment features of each audio segment to obtain the fused features of each audio segment, specifically includes the following: inputting each audio segment and its reference audio segment into the deformable convolutional layer of a pre-trained audio enhancement model for deformable convolution processing to obtain the alignment features between each audio segment and its reference audio segment; and concatenating the initial audio features and alignment features of each audio segment to obtain the concatenated features of each audio segment, which serve as the fused features of each audio segment.

[0103] The pre-trained audio enhancement model is primarily used to improve the audio quality of low-quality audio, thereby obtaining high-quality audio. Specifically, it is trained multiple times using sample audio from the financial system. In real-world scenarios, the network structure of the audio enhancement model is as follows: Figure 2 As shown.

[0104] The pre-trained audio enhancement model includes deformable convolutional layers, which consist of convolutional layers, concatenation layers, and deformable convolutional layers. These layers primarily perform deformable convolution processing on the current audio segment and its reference segment to obtain the audio relationship between them, thus yielding alignment features. In real-world scenarios, the network structure of the deformable convolutional layer is as follows: Figure 3 As shown.

[0105] Among them, convolutional layers refer to ordinary convolutional layers, which are mainly implemented through convolutional modules (conv) or convolutional neural networks, and are specifically used for feature extraction and feature optimization; concatenation layers are mainly implemented through concatenation modules (concat), and are specifically used for concatenating multiple features together; deformable convolutional layers are mainly implemented through deformable convolutional modules (Dconv) or deformable convolutional networks, and are specifically used for feature extraction and feature optimization.

[0106] Deformable convolution refers to a convolution kernel that adds an extra direction parameter to each element, so that the convolution kernel can expand to a large range during processing, i.e., a larger receptive field.

[0107] It should be noted that this application uses a deformable deep convolutional network to perform convolution processing on the audio features of audio segments, thereby improving the expressive power of convolution and thus better processing audio segments. Since the convolution kernel of ordinary convolution is square and the receptive field is also square, but when audio is generated, the sound object is usually not calm and waveless, and the noise background sound will also fluctuate with the volume of the sound. By adding an offset vector to each element of the square convolution kernel, each point in the convolution kernel is moved to the object, and then the weighted sum is performed, better feature extraction results can be obtained.

[0108] Among them, splicing features refer to the features obtained by splicing together the initial audio features and alignment features of an audio segment.

[0109] Specifically, the server inputs each audio segment and its reference audio segment into a pre-trained audio enhancement model. The deformable convolutional layer in the audio enhancement model performs alignment feature extraction on each audio segment and its reference audio segment to obtain alignment features between them. Feature extraction is then performed on each audio segment to obtain its initial audio features. Finally, the concatenation layer in the audio enhancement model concatenates the alignment features between each audio segment and its reference audio segment, along with the initial audio features of each audio segment, to obtain the concatenated features of each audio segment, which serve as the fusion features for each audio segment.

[0110] For example, see reference. Figure 3 The server inputs the current audio segment and the reference audio segment. Figure 3 In the network shown, through Figure 3 The network shown performs a series of feature extraction processes on the current audio segment and the reference audio segment, such as deformable convolution, to obtain the alignment features between the current audio segment and the reference audio segment.

[0111] In this embodiment, by performing deformable convolution processing on each audio segment and its reference audio segment, alignment features between each audio segment and its reference audio segment are obtained. These features are then concatenated with the initial audio features of each audio segment to obtain the fusion features of each audio segment. By obtaining the alignment features between each audio segment and its reference audio segment, as well as the initial audio features of each audio segment, it is beneficial to comprehensively consider both the alignment features and the initial audio features when enhancing the audio segments in the future, avoiding the omission of audio information. This results in higher audio quality in the reconstructed enhanced audio segments, thereby improving the audio enhancement effect.

[0112] In one embodiment, such as Figure 4 As shown, step S103 above involves feature extraction processing of the fusion features of each audio segment to obtain the target audio features of each audio segment. Specifically, this includes the following steps:

[0113] Step S401: Input the fused features of each audio segment into the first feature extraction layer for convolution processing to obtain the first audio features of each audio segment.

[0114] Step S402: Input the first audio feature of each audio segment into the second feature extraction layer for multiple feature extraction processes to obtain the second audio feature of each audio segment.

[0115] Step S403: The first and second audio features of each audio segment are superimposed and then input into the third feature extraction layer for multiple feature extraction processes to obtain the target audio features of each audio segment.

[0116] The pre-trained audio enhancement model includes a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer connected in sequence. These layers are primarily used to perform a series of feature extraction processes on each audio segment, continuously optimizing the audio features of each segment to obtain its deep audio features, thereby yielding the target audio features for each segment. It should be noted that the first, second, and third feature extraction layers are all used for feature extraction and feature optimization.

[0117] Among them, reference Figure 2 The first feature extraction layer includes a first convolutional layer, which is mainly implemented through a convolutional module or a convolutional neural network and is specifically used to extract features.

[0118] Among them, reference Figure 2 The second feature extraction layer comprises a second convolutional layer, a first dense block, a second dense block, a third dense block, a concatenation layer, a third convolutional layer, and a fourth convolutional layer, connected sequentially. The first and second dense blocks are also connected to the concatenation layer. The second, third, and fourth convolutional layers are all implemented using convolutional modules or convolutional neural networks, specifically used for feature extraction. The first, second, and third dense blocks are all implemented using residual dense blocks or densely connected networks, mainly used for feature extraction and optimization. The concatenation layer is mainly implemented using connection modules, specifically used to concatenate multiple features together, such as concatenating the output audio features of the first, second, and third dense blocks.

[0119] Among them, reference Figure 2 The third feature extraction layer consists of a fifth convolutional layer, a residual layer, a sixth convolutional layer, and a reconstruction layer connected in sequence. The fifth and sixth convolutional layers are implemented using convolutional modules or convolutional neural networks, specifically for feature extraction. The residual layer is mainly implemented using residual modules (resblocks) or residual networks, specifically for feature extraction and optimization. The reconstruction layer is mainly implemented using a reconstruction network, specifically for feature optimization.

[0120] The first audio feature refers to the audio feature obtained after convolutional processing of the fused features of the audio segment. The second audio feature refers to the audio feature obtained after multiple feature extraction processes on the first audio feature of the audio segment through the second convolutional layer, the first dense block, the second dense block, the third dense block, the splicing layer, the third convolutional layer, and the fourth convolutional layer. The target audio feature refers to the audio feature obtained after superimposing the first and second audio features of the audio segment and then performing multiple feature extraction processes through the fifth convolutional layer, the residual layer, the sixth convolutional layer, and the reconstruction layer. It should be noted that the feature levels of the first audio feature, the second audio feature, and the target audio feature increase progressively.

[0121] Specifically, refer to Figure 2Taking the current audio segment as an example, the server inputs the fused features of the current audio segment into the first feature extraction layer. The first feature extraction layer performs convolution processing on the fused features of the current audio segment to obtain the first audio feature of the current audio segment. The first audio feature of the current audio segment is then input into the second convolutional layer, where it performs convolution processing to obtain the audio feature output by the second convolutional layer. The audio feature output by the second convolutional layer is then input into a first dense block, where it performs feature extraction processing on the audio feature output by the second convolutional layer to obtain the audio feature output by the first dense block. The audio features output from the dense blocks are input into the concatenation layer. Simultaneously, the audio features output from the first dense block are input into the second dense block. The second dense block performs feature extraction on the audio features output from the first dense block to obtain the audio features output from the second dense block. The audio features output from the second dense block are then input into the concatenation layer. Simultaneously, the audio features output from the second dense block are input into the third dense block. The third dense block performs feature extraction on the audio features output from the second dense block to obtain the audio features output from the third dense block. The audio features output from the third dense block are then input into the concatenation layer. The concatenation layer then processes the audio features output from the first and second dense blocks... The audio features and the audio features output from the third dense block are concatenated to obtain the audio features output from the concatenation layer. The audio features output from the concatenation layer are then input into the third convolutional layer, where they are convolved to obtain the audio features output from the third convolutional layer. These audio features are then input into the fourth convolutional layer, where they are convolved to obtain the audio features output from the fourth convolutional layer, which serve as the second audio feature of the current audio segment. Finally, the audio features output from the first convolutional layer (i.e., the first audio feature of the current audio segment) are combined with the audio features output from the fourth convolutional layer. The audio features output from each layer are stacked and then fed into the fifth convolutional layer for convolution processing to obtain the audio features output from the fifth convolutional layer. The audio features output from the fifth convolutional layer are then fed into a residual layer, where residual processing is applied to the audio features output from the fifth convolutional layer to obtain the audio features output from the residual layer. The audio features output from the residual layer are then fed into a sixth convolutional layer, where convolution processing is applied to the audio features output from the residual layer to obtain the audio features output from the sixth convolutional layer. Finally, the audio features output from the sixth convolutional layer are fed into a reconstruction layer for feature extraction processing to obtain the audio features output from the reconstruction layer, which serve as the target audio features for the current audio segment.

[0122] In this embodiment, when performing feature extraction processing on each audio segment, the first feature extraction layer, the second feature extraction layer, and the third feature extraction layer, which are sequentially connected in the pre-trained audio enhancement model, are used to perform a series of feature extraction processes on the fusion features of each audio segment. This helps to continuously optimize the audio features of each audio segment, thereby making the final determined target audio features more accurate, and thus improving the audio enhancement effect of the enhanced audio segment reconstructed through the target audio features.

[0123] In one embodiment, such as Figure 5 As shown, the audio enhancement method provided in this application also includes a training step for a pre-trained audio enhancement model, specifically including the following steps:

[0124] Step S501: Obtain the sample audio of the financial system and the actual enhanced audio of the sample audio.

[0125] Step S502: The sample audio is segmented to obtain multiple sample audio segments.

[0126] Step S503: Input each sample audio segment and its reference sample audio segment into the audio enhancement model to be trained to obtain the predicted enhanced audio segment for each sample audio segment.

[0127] Step S504: Combine the predicted enhanced audio segments of each sample audio segment to obtain the predicted enhanced audio of the sample audio.

[0128] Step S505: Based on the predicted enhanced audio and the actual enhanced audio of the sample audio, train the audio enhancement model to be trained to obtain the trained audio enhancement model, which serves as the pre-trained audio enhancement model.

[0129] Among them, sample audio refers to audio with poor audio quality, actual enhanced audio refers to audio with higher audio quality corresponding to the sample audio, and predicted enhanced audio refers to the reconstructed audio of the sample audio.

[0130] Here, a sample audio segment refers to the audio segment obtained after segmenting the sample audio. A reference sample audio segment refers to an adjacent sample audio segment, such as the preceding or following sample audio segment. A predicted enhanced audio segment is the reconstructed audio segment corresponding to the sample audio segment, whose audio quality is higher than that of the sample audio segment.

[0131] The audio enhancement model to be trained also consists of deformable convolutional layers, a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer, with the following structure: Figure 2 As shown.

[0132] Specifically, the server collects the original audio recorded by the financial system as the original audio; it performs audio attenuation processing on the original audio, such as generating low-quality audio of the original audio through a neural network, as the sample audio; and it performs audio enhancement processing on the original audio, such as removing abnormal frequencies, spikes, background noise and other interference elements, to obtain high-quality audio of the original audio, which is used as the actual enhanced audio of the sample audio. Next, the server inputs the sample audio into a pre-trained lossless audio segmentation model for lossless segmentation, resulting in multiple sample audio segments. From these multiple sample audio segments, a reference sample audio segment is identified for each segment. Each sample audio segment and its reference sample audio segment are then input into an audio enhancement model to be trained. Through deformable convolutional layers in the audio enhancement model, alignment feature extraction is performed on each sample audio segment and its reference sample audio segment, yielding alignment features between them. Feature extraction is then performed on each sample audio segment to obtain initial audio features. Finally, through a concatenation layer in the audio enhancement model, the alignment features between each sample audio segment and its reference sample audio segment, as well as the initial audio features of each sample audio segment, are further refined. The features are concatenated to obtain the concatenated features of each audio sample segment, which are used as the fusion features of each audio sample segment. The fusion features of each audio sample segment are then input into the first feature extraction layer of the audio enhancement model to be trained. The first feature extraction layer performs convolution processing on the fusion features of each audio sample segment to obtain the first audio feature of each audio sample segment. The first audio feature of each audio sample segment is then input into the second feature extraction layer of the audio enhancement model to be trained. The second feature extraction layer performs multiple feature extraction processing on the first audio feature of each audio sample segment to obtain the second audio feature of each audio sample segment. The first audio feature and the second audio feature of each audio sample segment are then superimposed and input into the third feature extraction layer of the audio enhancement model to be trained. The third feature extraction layer performs multiple feature extraction processing on the superimposed result of the first audio feature and the second audio feature to obtain the target audio feature of each audio sample segment. Next, the server overlays the target audio features of each sample audio segment with each sample audio segment to obtain the reconstructed audio segment of each sample audio segment, which serves as the enhanced audio segment of each sample audio segment. Then, according to the order of arrangement of each sample audio segment, the predicted enhanced audio segments of each sample audio segment are combined to obtain the combined audio, which serves as the predicted enhanced audio of the sample audio.Finally, the server calculates the target loss value based on the predicted and actual enhanced audio of the sample audio, combined with the loss function. The audio enhancement model to be trained is then trained based on the target loss value until the target loss value obtained by the trained audio enhancement model is less than or equal to a preset threshold. The trained audio enhancement model is then used as the completed audio enhancement model, thus obtaining the pre-trained audio enhancement model.

[0133] For example, if the target loss value is greater than a preset threshold, the server adjusts the model parameters of the audio enhancement model to be trained according to the target loss value and repeats the above training process, such as steps S503 to S505, to retrain the audio enhancement model after the model parameters are adjusted, until the target loss value obtained from the trained audio enhancement model is less than or equal to the preset threshold, then training stops. The trained audio enhancement model at this time is the audio enhancement model that has been trained.

[0134] In this embodiment, the audio enhancement model to be trained is trained multiple times using sample audio from the financial system and the actual enhanced audio of the sample audio to obtain a trained audio enhancement model. This helps to improve the accuracy of the target audio features output by the audio enhancement model, thereby improving the audio quality of the enhanced audio segments obtained by superimposing the target audio features, and further improving the audio quality of the enhanced audio composed of the various enhanced audio segments, thus further improving the audio enhancement effect.

[0135] In one embodiment, step S505 above, which trains the audio enhancement model to be trained based on the predicted and actual enhanced audio of the sample audio to obtain a trained audio enhancement model, specifically includes the following: determining the reconstruction loss value and the perceptual loss value based on the predicted and actual enhanced audio of the sample audio; fusing the reconstruction loss value and the perceptual loss value to obtain the target loss value; training the audio enhancement model to be trained based on the target loss value and a preset channel pruning method until the training termination condition is met; and using the trained audio enhancement model that has met the training termination condition as the trained audio enhancement model.

[0136] The reconstruction loss value refers to the mean squared error loss value, which is used to maintain the structural consistency between the predicted and actual enhanced audio. The perceptual loss value is used to enhance the details of the audio. The target loss value is obtained by weighted summation of the reconstruction loss value and the perceptual loss value.

[0137] Among them, the training termination conditions are that the target loss value is less than or equal to the preset threshold, or the total number of training iterations reaches the preset number of training iterations.

[0138] Among them, the preset channel pruning method is mainly used to compress the network structure of the audio enhancement model. For example, it can perform a lightweight processing on the audio enhancement model, reduce the number of parameters and computation, so that the audio enhancement model can be small and fast while maintaining quality, so as to facilitate real-time audio enhancement.

[0139] Specifically, the server obtains the squared difference between the target enhanced audio and the actual enhanced audio for each sample audio. Then, it sums the squared differences between the target enhanced audio and the actual enhanced audio for each sample audio, calculates the average, and uses this average as the reconstruction loss value. Next, the server obtains the audio features of the target enhanced audio and the actual enhanced audio for each sample audio, and calculates the perceptual loss value by combining them with the perceptual loss function. The reconstruction loss value and the perceptual loss value are then weighted and summed to obtain the target loss value. Finally, based on the target loss value, the server adjusts the model parameters of the audio enhancement model to be trained and repeats the above training process to refine the audio enhancement model with adjusted model parameters. The training process is repeated. Simultaneously, during model training, the server obtains the channel scores from the audio enhancement model and prunes channels with lower scores (i.e., neurons on channels with lower scores are no longer connected). For example, a scaling factor is introduced for each channel, regularization is applied to the scaling factor, the channel scores are evaluated using the weights of the normalization layer, and then the scores are thresholded to remove channels with lower scores. The model is trained using these two processes until a training termination condition is met, such as a target loss value obtained from the trained audio enhancement model being less than or equal to a preset threshold. Finally, the trained audio enhancement model that has met the training termination condition is considered the completed audio enhancement model.

[0140] For example, the server first collects the raw audio data; it then performs data cleaning on the raw audio data, such as removing abnormal frequencies, spikes, background noise, and other interfering elements, to obtain high-quality audio data, which serves as the label; the raw audio data is then processed by a neural network to obtain low-quality audio data, which serves as the input data; the weighted sum of mean squared error loss and perceptual loss is used as the target loss function, and the training data (including the input data and labels) is used to train the audio enhancement model to be trained until the accuracy of the trained audio enhancement model reaches a threshold, at which point training stops, thus obtaining the trained audio enhancement model.

[0141] Furthermore, during model training, sparsification of network channels can reduce the size and computational memory required for the audio enhancement model. This reduces the number of computational operations without affecting model accuracy, which is significant for real-time audio enhancement. Taking VGG (a classic convolutional neural network) as an example, using relevant channel pruning can reduce the model size to 1 / 20 of its original size and the number of computational operations to 1 / 5. Specifically, optimization is performed on the basic normalization layer within the module. The normalization layer is mainly used for mean normalization, with the formula y = Wx + B, where x is the input, y is the output, W is the scaling factor, and B is the offset. Both W and B are parameters that need to be learned. Channel pruning involves applying regularization to the scaling factor, using the weights of the normalization layer to evaluate the scores of the input channels, and applying a threshold filter to the scores—mean filters out neurons on channels with lower scores—by performing this pruning operation layer by layer, thus compressing the model structure. It's important to note that the improvements here pertain to the score. First, B is fixed, and the content requiring compensation is moved to the subsequent fine-tuning section. This fine-tuning module is not pruned, and most of the network can reduce parameters and computation by half when passing through the normalization layer. Second, L2 regularization is used instead of L1 regularization; the square root of L2 regularization can more quickly eliminate channels with smaller weights, which is beneficial for training convergence. The last basic block and the final reconstruction module are not pruned and are used to accommodate parameter fine-tuning.

[0142] In this embodiment, reconstruction loss and perceptual loss are determined based on the predicted and actual enhanced audio of the sample audio. These two losses are then fused to obtain the target loss. Finally, the audio enhancement model is trained using the target loss and a preset channel pruning method to obtain the trained audio enhancement model. This process, utilizing reconstruction and perceptual loss values ​​to train the audio enhancement model multiple times during training, improves the audio enhancement effect of the trained model. Simultaneously, pruning the model structure compresses the model, minimizing its complexity and computational load without affecting accuracy, thus improving audio enhancement efficiency.

[0143] In one embodiment, step S105, which combines the enhanced audio segments of each audio segment to obtain the enhanced audio of the audio to be processed, specifically includes the following: inputting the enhanced audio segments of each audio segment into a pre-trained audio quality recognition model to obtain the audio quality prediction result of the enhanced audio segments of each audio segment; determining the arrangement order of each audio segment if the audio quality prediction result of the enhanced audio segments of each audio segment meets the preset audio quality conditions; and combining the enhanced audio segments of each audio segment according to the arrangement order to obtain the enhanced audio of the audio to be processed.

[0144] Among them, the pre-trained audio quality recognition model is mainly used to identify the audio quality of audio. Specifically, it is obtained by training the neural network model multiple times using sample audio from the financial system.

[0145] The audio quality prediction result refers to the audio quality level or audio quality score. The preset audio quality conditions refer to the preset audio quality level or preset audio quality score.

[0146] The order of audio segments refers to their chronological order; for example, the audio segment from the first to the second second is placed before the audio segment from the second to the third second.

[0147] Specifically, the server inputs the enhanced audio segments of each audio segment into a pre-trained audio quality recognition model. The model predicts the audio quality of each enhanced audio segment, obtaining a predicted audio quality result. For example, it predicts the probability of each enhanced audio segment belonging to various audio quality levels, and identifies the audio quality level with the highest probability as the predicted audio quality result. Next, the server determines whether the predicted audio quality result of each enhanced audio segment meets preset audio quality conditions. For example, it determines whether the audio quality level of each enhanced audio segment is greater than a preset audio quality level. If so, it determines that the predicted audio quality result of each enhanced audio segment meets the preset audio quality conditions, obtains the arrangement order of each audio segment, and combines the enhanced audio segments of each audio segment according to this arrangement order to obtain a combined audio, which serves as the enhanced audio for the audio to be processed.

[0148] Furthermore, if the audio quality prediction result of the enhanced audio segment does not meet the preset audio quality conditions, such as if the audio quality level of the enhanced audio segment is less than or equal to the preset audio quality level, then the enhanced audio segment is re-inputted into the pre-trained audio enhancement model for audio enhancement processing until the final audio quality prediction result of the enhanced audio segment meets the preset audio quality conditions.

[0149] In this embodiment, the enhanced audio segments of each audio segment are input into a pre-trained audio quality recognition model to obtain the audio quality prediction result of the enhanced audio segments of each audio segment. If the audio quality prediction result of the enhanced audio segments of each audio segment meets the preset audio quality conditions, the enhanced audio segments of each audio segment are combined according to the order of each audio segment to obtain the enhanced audio of the audio to be processed. In this way, when combining audio, the audio quality of the enhanced audio segments of each audio segment is first verified by the pre-trained audio quality recognition model to see if the audio quality meets the conditions. Only when the conditions are met are the enhanced audio segments combined together, which helps to ensure that the audio quality of each enhanced audio segment in the enhanced audio is high, and further improves the audio enhancement effect.

[0150] In one embodiment, step S105, after combining the enhanced audio segments of each audio segment to obtain the enhanced audio of the audio to be processed, further includes the following: extracting the target audio features of the enhanced audio; adding the target audio features of the enhanced audio to the audio to be processed to obtain the target enhanced audio of the audio to be processed.

[0151] The target audio features for enhanced audio refer to the deep audio features of the enhanced audio. The audio quality of the target enhanced audio is higher than that of the enhanced audio.

[0152] Specifically, the server performs convolution and residual processing on the enhanced audio to further optimize its audio features, obtaining deep audio features of the enhanced audio, which serve as the target audio features for the enhanced audio. The target audio features of the enhanced audio are then superimposed on the audio to be processed to obtain the target enhanced audio of the audio to be processed.

[0153] For example, the server inputs the enhanced audio into the third feature extraction layer for feature extraction processing to obtain the target audio features of the enhanced audio, and then superimposes them with the audio to be processed to obtain the reconstructed audio, which serves as the target enhanced audio for the audio to be processed.

[0154] In this embodiment, the target audio features of the enhanced audio are first extracted, and then the target audio features of the enhanced audio are added to the audio to be processed to obtain the target enhanced audio of the audio to be processed. In this way, after obtaining the enhanced audio of the audio to be processed, the target audio features of the enhanced audio are extracted again and superimposed on the audio to be processed, which is beneficial to perform enhanced audio processing again and further improve the audio enhancement effect.

[0155] In one embodiment, such as Figure 6As shown, another audio enhancement method is provided. Taking the application of this method to a server as an example, the steps include:

[0156] Step S601: The audio to be processed in the financial system is segmented to obtain multiple audio segments.

[0157] Step S602: Input each audio segment and its reference audio segment into the deformable convolutional layer of the pre-trained audio enhancement model for deformable convolution processing to obtain the alignment features between each audio segment and its reference audio segment.

[0158] Step S603: Using a pre-trained audio enhancement model, the initial audio features and alignment features of each audio segment are spliced ​​together to obtain the spliced ​​features of each audio segment, which serve as the fusion features of each audio segment.

[0159] Step S604: Input the fused features of each audio segment into the first feature extraction layer of the pre-trained audio enhancement model for convolution processing to obtain the first audio features of each audio segment.

[0160] Step S605: Input the first audio feature of each audio segment into the second feature extraction layer of the pre-trained audio enhancement model for multiple feature extraction processes to obtain the second audio feature of each audio segment.

[0161] Step S606: After superimposing the first and second audio features of each audio segment, the result is input into the third feature extraction layer of the pre-trained audio enhancement model for multiple feature extraction processes to obtain the target audio features of each audio segment.

[0162] Step S607: Add the target audio features of each audio segment to each audio segment to obtain the enhanced audio segment of each audio segment.

[0163] Step S608: Input the enhanced audio segment of each audio segment into the pre-trained audio quality recognition model to obtain the audio quality prediction result of the enhanced audio segment of each audio segment; if the audio quality prediction result of the enhanced audio segment of each audio segment meets the preset audio quality conditions, determine the arrangement order of each audio segment.

[0164] Step S609: Combine the enhanced audio segments of each audio segment according to the arrangement order to obtain the enhanced audio of the audio to be processed.

[0165] Step S610: Extract the target audio features of the enhanced audio; add the target audio features of the enhanced audio to the audio to be processed to obtain the target enhanced audio of the audio to be processed.

[0166] The audio enhancement method provided in the above embodiments first divides the audio to be processed into multiple audio segments during audio enhancement. It then utilizes the alignment features between each audio segment and its reference audio segment to assist in obtaining the target audio features of each audio segment. These target audio features are then used to enhance each audio segment, resulting in an enhanced audio segment for each segment. Finally, the enhanced audio segments are combined to obtain the enhanced audio of the audio to be processed. In other words, by enhancing audio by segmenting it into segments and then combining the enhanced audio segments, the information of each audio segment can be fully considered, avoiding the omission of key audio information and thus improving the audio enhancement effect. Furthermore, when extracting features from each audio segment, the alignment features between each audio segment and its reference audio segment are comprehensively considered, which helps to further optimize the audio features of each segment, making the reconstructed enhanced audio segments more accurate. This, in turn, makes the enhanced audio obtained by combining the enhanced audio segments more accurate, further improving the audio enhancement effect. Simultaneously, it avoids the drawback of enhancing audio through filtering, which easily leads to the omission of key audio information and results in poor audio enhancement.

[0167] In one embodiment, to more clearly illustrate the audio enhancement method provided in this application, the following detailed description uses a specific embodiment. In one embodiment, refer to... Figure 2 This application also provides an audio enhancement method. An audio enhancement network is constructed by referencing the structure of a video enhancement network. The constructed audio enhancement network is then trained to obtain a trained audio enhancement network. Finally, the trained audio enhancement network is placed at the receiving end to reconstruct the audio data transmitted through the audio compression transmission network, thereby outputting a high-quality audio file. This solves the problem of poor audio enhancement effect in traditional audio enhancement methods. The specific details are as follows:

[0168] refer to Figure 2 The server first segments the audio to be processed, obtaining multiple audio segments. Then, for the current audio segment, the server inputs the current audio segment, the preceding audio segment, and the following audio segment into a pre-trained audio enhancement model for a series of processing steps to obtain a reconstructed audio segment. The same method is used to obtain the reconstructed audio segments for each individual audio segment. Finally, the server combines the reconstructed audio segments in chronological order to obtain a high-quality reconstructed audio file.

[0169] The audio enhancement method provided in the above embodiments constructs an audio enhancement network based on the structure of a video enhancement network, and performs lightweight processing on the audio enhancement network to reduce the number of parameters and computational load. This allows the audio enhancement network to maintain model accuracy while being small and fast, facilitating real-time audio enhancement. Furthermore, after training the constructed audio enhancement network, enhancement processing is performed on the audio segments to be processed, and the alignment features between adjacent audio segments are considered, which further improves the audio enhancement effect.

[0170] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0171] Based on the same inventive concept, this application also provides an audio enhancement apparatus for implementing the audio enhancement method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more audio enhancement apparatus embodiments provided below can be found in the limitations of the audio enhancement method described above, and will not be repeated here.

[0172] In one embodiment, such as Figure 7 As shown, an audio enhancement device is provided, including: an audio segmentation module 710, a feature fusion module 720, a feature extraction module 730, a feature addition module 740, and an audio combination module 750, wherein:

[0173] The audio segmentation module 710 is used to segment the audio to be processed in the financial system to obtain multiple audio segments.

[0174] The feature fusion module 720 is used to obtain the alignment features between each audio segment and its reference audio segment, and to fuse the initial audio features and alignment features of each audio segment to obtain the fused features of each audio segment.

[0175] The feature extraction module 730 is used to perform feature extraction processing on the fusion features of each audio segment to obtain the target audio features of each audio segment.

[0176] The feature addition module 740 is used to add the target audio features of each audio segment to each audio segment, so as to obtain the enhanced audio segment of each audio segment.

[0177] The audio combination module 750 is used to combine the enhanced audio segments of each audio segment to obtain the enhanced audio of the audio to be processed.

[0178] In one embodiment, the feature fusion module 720 is further configured to input each audio segment and its reference audio segment into a deformable convolutional layer in a pre-trained audio enhancement model for deformable convolution processing to obtain alignment features between each audio segment and its reference audio segment; and to concatenate the initial audio features and alignment features of each audio segment to obtain concatenated features of each audio segment, which serve as the fusion features of each audio segment.

[0179] In one embodiment, the pre-trained audio enhancement model further includes a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer connected in sequence. The first feature extraction layer includes a first convolutional layer. The second feature extraction layer includes a second convolutional layer, a first dense block, a second dense block, a third dense block, a splicing layer, a third convolutional layer, and a fourth convolutional layer connected in sequence. The first and second dense blocks are also connected to the splicing layer. The third feature extraction layer includes a fifth convolutional layer, a residual layer, a sixth convolutional layer, and a reconstruction layer connected in sequence.

[0180] The feature extraction module 730 is also used to input the fused features of each audio segment into the first feature extraction layer for convolution processing to obtain the first audio feature of each audio segment; input the first audio feature of each audio segment into the second feature extraction layer for multiple feature extraction processes to obtain the second audio feature of each audio segment; and after superimposing the first audio feature and the second audio feature of each audio segment, input them into the third feature extraction layer for multiple feature extraction processes to obtain the target audio feature of each audio segment.

[0181] In one embodiment, the audio enhancement apparatus provided in this application further includes a model training module, used to acquire sample audio from a financial system and actual enhanced audio of the sample audio; segment the sample audio to obtain multiple sample audio segments; input each sample audio segment and a reference sample audio segment of each sample audio segment into the audio enhancement model to be trained to obtain a predicted enhanced audio segment for each sample audio segment; combine the predicted enhanced audio segments of each sample audio segment to obtain the predicted enhanced audio of the sample audio; and train the audio enhancement model to be trained based on the predicted enhanced audio and the actual enhanced audio of the sample audio to obtain a trained audio enhancement model, which serves as the pre-trained audio enhancement model.

[0182] In one embodiment, the model training module is further configured to determine the reconstruction loss value and the perceptual loss value based on the predicted enhanced audio and the actual enhanced audio of the sample audio; fuse the reconstruction loss value and the perceptual loss value to obtain the target loss value; train the audio enhancement model to be trained according to the target loss value and the preset channel pruning method until the training termination condition is met; and take the trained audio enhancement model that has met the training termination condition as the trained audio enhancement model.

[0183] In one embodiment, the audio combination module 750 is further configured to input the enhanced audio segments of each audio segment into a pre-trained audio quality recognition model to obtain the audio quality prediction result of the enhanced audio segments of each audio segment; if the audio quality prediction result of the enhanced audio segments of each audio segment meets the preset audio quality conditions, determine the arrangement order of each audio segment; and combine the enhanced audio segments of each audio segment according to the arrangement order to obtain the enhanced audio of the audio to be processed.

[0184] In one embodiment, the audio enhancement device provided in this application further includes an audio enhancement module for extracting target audio features of the enhanced audio; adding the target audio features of the enhanced audio to the audio to be processed to obtain the target enhanced audio of the audio to be processed.

[0185] Each module in the aforementioned audio enhancement device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0186] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data such as enhanced audio clips and enhanced audio. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an audio enhancement method.

[0187] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0188] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0189] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0190] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0191] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0192] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0193] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An audio enhancement method, characterized in that, The method includes: The audio to be processed in the financial system is segmented to obtain multiple audio segments of the audio to be processed; Each audio segment and its reference audio segment are input into a deformable convolutional layer in a pre-trained audio enhancement model for deformable convolution processing to obtain alignment features between each audio segment and its reference audio segment. The initial audio features and alignment features of each audio segment are spliced ​​together to obtain the splicing features of each audio segment, which serve as the fusion features of each audio segment. The reference audio segment of each audio segment is the adjacent audio segment of that audio segment. Specifically, the reference audio segment of the first audio segment is its next audio segment, the reference audio segment of the last audio segment is its previous audio segment, and the reference audio segments of the middle audio segments are its preceding and following audio segments. The alignment features are used to represent the audio relationship between the audio segment and the reference audio segment of the audio segment. The fusion features of each audio segment are processed by feature extraction to obtain the target audio features of each audio segment; The target audio features of each audio segment are added to each audio segment to obtain the enhanced audio segment of each audio segment; The enhanced audio segments of each audio segment are combined to obtain the enhanced audio of the audio to be processed.

2. The method according to claim 1, characterized in that, The pre-trained audio enhancement model further includes a first feature extraction layer, a second feature extraction layer, and a third feature extraction layer connected in sequence. The first feature extraction layer includes a first convolutional layer. The second feature extraction layer includes a second convolutional layer, a first dense block, a second dense block, a third dense block, a splicing layer, a third convolutional layer, and a fourth convolutional layer connected in sequence. The first dense block and the second dense block are also connected to the splicing layer. The third feature extraction layer includes a fifth convolutional layer, a residual layer, a sixth convolutional layer, and a reconstruction layer connected in sequence. The step of performing feature extraction processing on the fusion features of each audio segment to obtain the target audio features of each audio segment includes: The fused features of each audio segment are input into the first feature extraction layer for convolution processing to obtain the first audio features of each audio segment; The first audio feature of each audio segment is input into the second feature extraction layer for multiple feature extraction processes to obtain the second audio feature of each audio segment. The first and second audio features of each audio segment are superimposed and then input into the third feature extraction layer for multiple feature extraction processes to obtain the target audio features of each audio segment.

3. The method according to claim 2, characterized in that, The pre-trained audio enhancement model is trained in the following manner: Acquire sample audio from the financial system and the actual enhanced audio of the sample audio; The sample audio is segmented to obtain multiple sample audio segments; Each sample audio segment and its reference sample audio segment are input into the audio enhancement model to be trained to obtain the predicted enhanced audio segment for each sample audio segment. The predicted enhanced audio segments of each sample audio segment are combined to obtain the predicted enhanced audio of the sample audio. Based on the predicted enhanced audio and the actual enhanced audio of the sample audio, the audio enhancement model to be trained is trained to obtain the trained audio enhancement model, which is used as the pre-trained audio enhancement model.

4. The method according to claim 3, characterized in that, The step of training the audio enhancement model to be trained based on the predicted enhanced audio and the actual enhanced audio of the sample audio to obtain the trained audio enhancement model includes: Based on the predicted enhanced audio and the actual enhanced audio of the sample audio, determine the reconstruction loss value and the perceptual loss value; The reconstruction loss value and the perception loss value are fused together to obtain the target loss value; The audio enhancement model to be trained is trained according to the target loss value and the preset channel pruning method until the training termination condition is met. The trained audio enhancement model that meets the training termination condition is considered the completed audio enhancement model.

5. The method according to claim 1, characterized in that, The step of combining the enhanced audio segments of each audio segment to obtain the enhanced audio of the audio to be processed includes: The enhanced audio segments of each audio segment are input into a pre-trained audio quality recognition model to obtain the audio quality prediction results of the enhanced audio segments of each audio segment; If the audio quality prediction result of the enhanced audio segment of each audio segment meets the preset audio quality conditions, the order of arrangement of each audio segment is determined; According to the stated arrangement, the enhanced audio segments of each audio segment are combined to obtain the enhanced audio of the audio to be processed.

6. The method according to any one of claims 1 to 5, characterized in that, After combining the enhanced audio segments of each audio segment to obtain the enhanced audio of the audio to be processed, the process further includes: Extract the target audio features of the enhanced audio; The target audio features of the enhanced audio are added to the audio to be processed to obtain the target enhanced audio of the audio to be processed.

7. An audio enhancement device, characterized in that, The device includes: The audio segmentation module is used to segment the audio to be processed in the financial system to obtain multiple audio segments of the audio to be processed. The feature fusion module is used to input each audio segment and its reference audio segment into a deformable convolutional layer in a pre-trained audio enhancement model for deformable convolution processing to obtain alignment features between each audio segment and its reference audio segment. The initial audio features and alignment features of each audio segment are then concatenated to obtain concatenated features for each audio segment, which serve as the fusion features of each audio segment. The reference audio segments for each audio segment are its adjacent audio segments, where the reference audio segment for the first audio segment is its next audio segment, the reference audio segment for the last audio segment is its previous audio segment, and the reference audio segments for the middle audio segments are its preceding and following audio segments. The alignment features are used to represent the audio relationship between an audio segment and its reference audio segment. The feature extraction module is used to perform feature extraction processing on the fusion features of each audio segment to obtain the target audio features of each audio segment; The feature addition module is used to add the target audio features of each audio segment to each audio segment to obtain the enhanced audio segment of each audio segment; An audio combination module is used to combine the enhanced audio segments of each audio segment to obtain the enhanced audio of the audio to be processed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images