Audio enhancement method, apparatus, device, computer-readable medium, and program product

Abstract

Embodiments of the present disclosure disclose an audio enhancement method, device, equipment, computer readable medium and program product. A specific implementation of the method comprises: obtaining a to-be-processed audio signal and a reference signal for a target scene; preprocessing the audio signal and the reference signal to obtain preprocessed audio feature information; inputting the preprocessed audio feature information into a pre-trained audio enhancement model to obtain each enhanced audio corresponding to different audio enhancement tasks in the target scene, wherein the audio enhancement model supports processing of different types of audio enhancement tasks. The implementation is related to audio enhancement, and based on the high speech enhancement performance of the audio enhancement model, different audio enhancement requirements in the target scene can be efficiently completed.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more particularly to audio enhancement methods, apparatus, devices, computer-readable media, and program products. Background Technology

[0002] Currently, different audio application scenarios often correspond to different audio enhancement requirements, and how to achieve efficient audio processing for these different requirements has become one of the current research directions. The common approach to handling different audio enhancement requirements is to employ different audio enhancement methods (e.g., AI model-based processing methods, signal processing-based methods, etc.) for each specific requirement.

[0003] However, when using the above method, the following technical problems often arise: The limited allocation of computing resources for audio applications, coupled with the need for different enhancement methods to meet varying audio enhancement requirements, exponentially increases the computational load and the number of parameters, leading to a strain on computing resources. In some cases, the lack of computing resources can even prevent the real-time processing of certain audio enhancement needs, resulting in audio output failures. Summary of the Invention

[0004] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0005] Some embodiments of this disclosure provide audio enhancement methods, apparatuses, electronic devices, computer-readable media, and program products to address the technical problems mentioned in the background section above.

[0006] In a first aspect, some embodiments of this disclosure provide an audio enhancement method, including: acquiring an audio signal to be processed and a reference signal for a target scene; preprocessing the audio signal and the reference signal to obtain preprocessed audio feature information; inputting the preprocessed audio feature information into a pre-trained audio enhancement model to obtain various enhanced audios corresponding to different audio enhancement tasks in the target scene, wherein the audio enhancement model supports processing different types of audio enhancement tasks.

[0007] Optionally, the above-mentioned preprocessing of the audio signal and the reference signal to obtain preprocessed audio feature information includes: performing frequency domain transformation on the audio signal and the reference signal to obtain first frequency domain information and second frequency domain information; and concatenating the first frequency domain information and the second frequency domain information to obtain preprocessed audio feature information.

[0008] Optionally, the aforementioned audio enhancement model includes: an audio feature extraction layer, a temporal feature extraction layer, a mask generation layer, and an output layer; and the method of inputting the aforementioned preprocessed audio feature information into the pre-trained audio enhancement model to obtain various enhanced audios corresponding to different audio enhancement tasks in the aforementioned target scene includes: inputting the aforementioned preprocessed audio feature information into the aforementioned audio feature extraction layer to obtain audio feature information corresponding to the time domain and frequency domain; inputting the aforementioned audio feature information into the aforementioned temporal feature extraction layer to obtain temporal feature information; inputting the aforementioned temporal feature information into the aforementioned mask generation layer to obtain mask information; and inputting the aforementioned mask information and the aforementioned preprocessed audio feature information into the aforementioned output layer to obtain various enhanced audios.

[0009] Optionally, the above-mentioned audio enhancement model is trained through the following steps: acquiring target training data, wherein the target training data includes: audio data, reference data, and at least one audio enhancement data for the audio data; preprocessing the target training data to obtain preprocessed data; inputting the preprocessed data into an initial audio enhancement model to obtain at least one initial enhanced audio; determining at least one audio enhancement loss between the at least one audio enhancement data and the at least one initial enhanced audio; determining whether the initial audio enhancement model has been trained successfully based on the at least one audio enhancement loss; and, in response to determining that training is complete, identifying the initial audio enhancement model as an audio enhancement model.

[0010] Optionally, before inputting the preprocessed audio feature information into the pre-trained audio enhancement model to obtain the enhanced audio corresponding to different audio enhancement tasks in the target scene, the method further includes determining whether a dedicated model corresponding to the target scene exists in the model database; in response to determining that it does not exist, determining whether the target scene is a sub-scene; in response to determining that it is a sub-scene, determining whether a corresponding dedicated model exists in the higher-level scene corresponding to the target scene; and in response to determining that it exists, determining the audio enhancement model based on the dedicated model corresponding to the higher-level scene.

[0011] Optionally, determining the audio enhancement model based on the dedicated model corresponding to the upper-level scenario includes: obtaining an audio enhancement training dataset for the target scenario; determining the audio enhancement task information corresponding to each audio enhancement task for the target scenario; and updating the model parameters in the dedicated model corresponding to the upper-level scenario and corresponding to the audio enhancement task information according to the audio enhancement training dataset to obtain the audio enhancement model.

[0012] Optionally, the method further includes: responding to the existence of enhancement effect information in the enhancement effect information corresponding to each of the enhanced audios where the representation effect does not meet the corresponding effect conditions, generating at least one corresponding enhancement training data based on at least one enhancement effect information that does not meet the conditions; storing the at least one enhancement training data in a model database; responding to the fact that the amount of enhancement training data corresponding to the target audio enhancement task stored in the model database reaches the target amount of data, setting a ratio of the amount of training data corresponding to the target audio enhancement task and the other audio enhancement task sets, wherein the amount of training data corresponding to the target audio enhancement task is higher than the amount of training data corresponding to the other audio enhancement tasks, wherein the task set between the other audio enhancement task sets and the target audio enhancement task is the same as the different types of audio enhancement tasks; obtaining the corresponding enhancement training dataset from the model database according to the training data ratio; and retraining the audio enhancement model according to the enhancement training dataset to obtain the retrained audio enhancement model for online processing.

[0013] Secondly, some embodiments of this disclosure provide an audio enhancement apparatus, including: an acquisition unit configured to acquire an audio signal to be processed and a reference signal for a target scene; a processing unit configured to preprocess the audio signal and the reference signal to obtain preprocessed audio feature information; and a generation unit configured to input the preprocessed audio feature information into a pre-trained audio enhancement model to obtain various enhanced audios corresponding to different audio enhancement tasks in the target scene, wherein the audio enhancement model supports processing different types of audio enhancement tasks.

[0014] Optionally, the processing unit can be configured to: perform frequency domain transformation on the audio signal and the reference signal to obtain first frequency domain information and second frequency domain information; and concatenate the first frequency domain information and the second frequency domain information to obtain preprocessed audio feature information.

[0015] Optionally, the above audio enhancement model includes: an audio feature extraction layer, a temporal feature extraction layer, a mask generation layer, and an output layer; and the generation unit can be configured to: input the above preprocessed audio feature information into the above audio feature extraction layer to obtain audio feature information corresponding to the time domain and frequency domain; input the above audio feature information into the above temporal feature extraction layer to obtain temporal feature information; input the above temporal feature information into the above mask generation layer to obtain mask information; and input the above mask information and the above preprocessed audio feature information into the above output layer to obtain various enhanced audio.

[0016] Optionally, the apparatus further includes: determining whether a dedicated model corresponding to the target scene exists in the model database; in response to determining that it does not exist, determining whether the target scene is a sub-scene; in response to determining that it is a sub-scene, determining whether a corresponding dedicated model exists in the higher-level scene corresponding to the target scene; and in response to determining that it exists, determining the audio enhancement model based on the dedicated model corresponding to the higher-level scene.

[0017] Optionally, the apparatus further includes: acquiring an audio enhancement training dataset for the target scenario; determining information on each audio enhancement task corresponding to the target scenario; and updating the model parameters in the dedicated model corresponding to the upper-level scenario based on the audio enhancement training dataset, thereby obtaining an audio enhancement model.

[0018] Optionally, the apparatus further includes: responding to the existence of enhancement effect information in the enhancement effect information corresponding to each of the above-mentioned enhanced audios where the representation effect does not meet the corresponding effect conditions, generating at least one corresponding enhancement training data based on at least one enhancement effect information that does not meet the conditions; storing the at least one enhancement training data in a model database; responding to the fact that the amount of enhancement training data corresponding to the target audio enhancement task stored in the model database reaches the target amount of data, setting a ratio of the amount of training data corresponding to the target audio enhancement task and the other audio enhancement task sets, wherein the amount of training data corresponding to the target audio enhancement task is higher than the amount of training data corresponding to the other audio enhancement tasks, wherein the task set between the other audio enhancement task sets and the target audio enhancement task is the same as the different types of audio enhancement tasks; obtaining the corresponding enhancement training dataset from the model database according to the training data ratio; and retraining the audio enhancement model according to the enhancement training dataset to obtain the retrained audio enhancement model for online processing.

[0019] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0020] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0021] Fifthly, some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0022] The various embodiments disclosed above have the following beneficial effects: Through the audio enhancement methods of some embodiments of this disclosure, based on the high speech enhancement performance corresponding to the audio enhancement model, different audio enhancement requirements in the target scenario can be efficiently completed. Specifically, the reason for the low efficiency in fulfilling different audio enhancement requirements is that the allocation of computing resources for audio application scenarios is limited. Different audio enhancement requirements employ different enhancement methods, causing the computational load and parameter count to increase exponentially, leading to a shortage of computing resources. Even the lack of computing resources can result in some audio enhancement requirements not being processed in real time, causing audio output failures. Based on this, the audio enhancement methods of some embodiments of this disclosure first acquire the audio signal to be processed and the reference signal for the target scenario to obtain the audio content to be processed, facilitating subsequent audio content enhancement processing. Then, the audio signal and the reference signal are preprocessed to obtain preprocessed audio feature information, facilitating the extraction of time-domain and / or frequency-domain features, which is beneficial for the subsequent audio enhancement model to learn audio semantic content and for input into the audio enhancement model. Finally, the preprocessed audio feature information is input into the pre-trained audio enhancement model. This allows for efficient processing of various enhanced audio values ​​for different audio enhancement tasks in the target scenario with minimal computational resources. The audio enhancement model supports processing different types of audio enhancement tasks. In summary, using a single audio enhancement model enables efficient implementation of various audio enhancement needs, avoiding the computational complexity and resource constraints caused by requiring multiple implementation methods for different audio enhancement requirements. Attached Figure Description

[0023] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0024] Figure 1 This is a schematic diagram illustrating an application scenario of an audio enhancement method according to some embodiments of the present disclosure; Figure 2 This is a flowchart of some embodiments of the audio enhancement method according to the present disclosure; Figure 3 These are flowcharts of other embodiments of the audio enhancement method according to this disclosure; Figure 4 This is a schematic diagram of the structure of some embodiments of the audio enhancement device according to the present disclosure; Figure 5 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0025] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0026] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0027] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0028] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0029] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0030] Before performing any of the operations involving the collection, storage, or use of user personal information (such as audio signals) disclosed in this disclosure, the relevant organizations or individuals shall fulfill their obligations, including conducting personal information security impact assessments, informing personal information subjects, and obtaining prior authorization and consent from personal information subjects.

[0031] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0032] Figure 1 This is a schematic diagram of an application scenario of an audio enhancement method according to some embodiments of the present disclosure.

[0033] exist Figure 1In this application scenario, firstly, the electronic device 101 can acquire the audio signal 102 to be processed and the reference signal 103 for the target scenario. In this application scenario, the audio signal 102 can be "audio from a car intelligent device". Then, the electronic device 101 can preprocess the audio signal 102 and the reference signal 103 to obtain preprocessed audio feature information 104. In this application scenario, the preprocessed audio feature information 104 can be "feature information corresponding to the time domain and frequency domain". Finally, the electronic device 101 can input the preprocessed audio feature information 104 into a pre-trained audio enhancement model 105 to obtain various enhanced audios corresponding to different audio enhancement tasks in the target scenario. The audio enhancement model 105 supports the processing of different types of audio enhancement tasks. In this application scenario, the target scenario can be the audio output scenario of a car intelligent device. Different audio enhancement tasks can include: call scenarios and voice wake-up scenarios. Various enhanced audios can include: call enhanced audio 106 and voice wake-up enhanced audio 107.

[0034] It should be noted that the aforementioned electronic device 101 can be either hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the electronic device is software, it can be installed in the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0035] It should be understood that Figure 1 The number of electronic devices shown is merely illustrative. Any number of electronic devices can be used depending on the implementation requirements.

[0036] Continue to refer to Figure 2 The diagram illustrates a flow 200 of some embodiments of an audio enhancement method according to the present disclosure. The audio enhancement method includes the following steps: Step 201: Obtain the audio signal to be processed and the reference signal for the target scene.

[0037] In some embodiments, the entity executing the above-described audio enhancement method (e.g.) Figure 1The electronic device 101 shown can acquire the audio signal to be processed and the reference signal for the target scenario via a wired or wireless connection. The target scenario can be the current audio enhancement processing scenario. It should be noted that the target scenario is the voice processing scenario corresponding to a smart terminal device (e.g., a robot, a new energy vehicle, headphones, a tablet, etc.). For example, the target scenario can be the audio processing scenario corresponding to a car's smart cockpit. There is at least one corresponding audio enhancement requirement in the target scenario. For example, for the audio processing scenario corresponding to a car's smart cockpit, two tasks need to be performed simultaneously: a call and a voice wake-up (i.e., there is an audio enhancement requirement under the call task and an audio enhancement requirement under the voice wake-up task). The audio signal can be the audio content to be processed (i.e., to be enhanced). In the audio enhancement scenario, the reference signal is a key auxiliary signal that helps improve the quality of the target audio (e.g., noisy speech, echo-containing speech). The core function of the reference signal is to provide prior information or a reference signal for the audio enhancement algorithm to achieve more accurate noise suppression, echo cancellation, speech separation, or quality optimization. The essence of a reference signal is to provide "additional information" for audio enhancement algorithms, compensating for the lack of information in a single input signal (noisy speech). For example, the following effects can be achieved through a reference signal to meet different audio enhancement needs: estimating noise / echo components (e.g., multi-microphone noise reduction, AEC), preserving speech details (e.g., speaker feature-guided enhancement), and ensuring system consistency (e.g., multi-channel synchronization).

[0038] Step 202: Preprocess the above audio signal and the above reference signal to obtain preprocessed audio feature information.

[0039] In some embodiments, the execution entity may preprocess the audio signal and the reference signal to obtain preprocessed audio feature information. Preprocessing may involve modifying the signal content to suit the input of a subsequent audio enhancement model. The preprocessed audio feature information can characterize the fused audio semantic content of the audio signal and the reference signal. In practice, the preprocessed audio feature information may be in vector form.

[0040] As an example, firstly, the first time-domain feature information and the first frequency-domain feature information corresponding to the aforementioned audio signal are extracted. Then, the second time-domain feature information and the second frequency-domain feature information corresponding to the aforementioned reference signal are extracted. Next, feature extraction and fusion are performed on the first time-domain feature information and the first frequency-domain feature information to obtain first feature fusion information. Then, feature extraction and fusion are performed on the second time-domain feature information and the second frequency-domain feature information to obtain second feature fusion information. Finally, the first feature fusion information and the second feature fusion information are combined to obtain preprocessed audio feature information.

[0041] In some optional implementations of certain embodiments, the execution entity may preprocess the audio signal and the reference signal to obtain preprocessed audio feature information, including the following steps: The first step involves performing a frequency domain transformation on the aforementioned audio signal and the aforementioned reference signal to obtain first frequency domain information and second frequency domain information. The first frequency domain information characterizes the frequency domain features of the audio signal. In practice, the first frequency domain information can be in vector form. The second frequency domain information characterizes the frequency domain features of the reference signal. In practice, the second frequency domain information can also be in vector form.

[0042] As an example, the aforementioned execution entity can use the Short-Time Fourier Transform (STFT) method to perform frequency domain transformation on the aforementioned audio signal and the aforementioned reference signal to obtain first frequency domain information and second frequency domain information.

[0043] The second step involves concatenating the first and second frequency domain information to obtain preprocessed audio feature information. For example, if the information dimension corresponding to the first frequency domain information is [160, 3] and the information dimension corresponding to the second frequency domain information is [160, 1], then the information dimension corresponding to the preprocessed audio feature information is [160, 4].

[0044] Step 203: Input the preprocessed audio feature information into the pre-trained audio enhancement model to obtain the enhanced audio corresponding to different audio enhancement tasks in the target scene.

[0045] In some embodiments, the execution entity can input the preprocessed audio feature information into a pre-trained audio enhancement model to obtain various enhanced audios corresponding to different audio enhancement tasks in the target scenario. The audio enhancement model can be a neural network model that enhances audio signals to meet different audio enhancement requirements in the corresponding scenario (i.e., to meet different audio enhancement tasks). For example, the audio enhancement requirement for a call task might be to remove as much noise as possible. The audio enhancement requirement for a recognition or wake-up task might be to remove noise as much as possible without damaging the speech. The audio enhancement requirement for a broadcast or two-way call task might be separate echo cancellation. Different audio enhancement tasks have corresponding audio enhancement requirements. Each audio enhancement task has corresponding enhanced audio. Enhanced audio can be audio after enhancing the audio signal. For example, each enhanced audio can include: audio enhanced for a call task, audio enhanced for a recognition or wake-up task, and audio enhanced for a broadcast or two-way call task. For example, the audio enhancement model can be a multimodal large model. Here, a multimodal large model can be a large model with a conventional architecture that supports multimodal content input or output. In practice, a multimodal large model can be a model trained on audio enhancement training datasets based on various audio enhancement tasks. Audio enhancement training data can include: input audio and enhanced audio.

[0046] It should be noted that the audio enhancement model here can output enhanced audio for different enhancement needs. That is, the audio enhancement model corresponds to multiple output layers, and each output layer has a corresponding audio enhancement task. Each output layer outputs the corresponding enhanced audio.

[0047] In some optional implementations of certain embodiments, after step 203, the steps further include: The first step is to determine whether a dedicated model corresponding to the target scenario exists in the model database. The model database can be a database storing model-related data. In practice, model-related data can include various types of models and training datasets for different execution tasks. The dedicated model corresponding to the target scenario can be a model specifically designed to handle various audio enhancement requirements for that target scenario. In practice, the dedicated model can be a neural network model.

[0048] The second step, in response to the determination that the target scenario does not exist, is to determine whether the aforementioned target scenario is a sub-scenario. A sub-scenario can be a scenario within a parent scenario (i.e., a sub-scenario is a subordinate scenario of the parent scenario, and the parent scenario is the parent scenario of the sub-scenario). A parent scenario can have at least one sub-scenario. For example, a parent scenario can be an audio processing scenario under a smart home. A smart home has at least one smart device. Therefore, the corresponding at least one subordinate scenario can correspond to at least one audio processing sub-scenario corresponding to at least one smart device. For example, at least one audio processing sub-scenario can include: "audio processing scenario corresponding to smart lighting," "audio processing scenario corresponding to a smart refrigerator," and "audio processing scenario corresponding to a smart door lock." As another example, for an audio processing scenario corresponding to a car smart device under a parent scenario, the corresponding at least one audio processing sub-scenario can be the audio processing scenario corresponding to each smart module under the car smart device. In practice, the scenario and model association database can be used to query whether a target scenario is a sub-scenario. The scenario and model association database stores: scenario information corresponding to each scenario, a scenario association table showing the relationship between scenarios and corresponding dedicated models, and model information for each dedicated model. In practice, the scenario and model association database can be maintained in real time.

[0049] As an example, the aforementioned executing entity can determine whether the target scenario has a corresponding parent scenario, i.e., whether it is a sub-scenario, by querying the scenario association table.

[0050] The third step, in response to determining that it is a sub-scene, is to determine whether a corresponding dedicated model exists for the higher-level scene corresponding to the aforementioned target scene. The dedicated model for the higher-level scene can be a unique model that performs audio enhancement for each audio enhancement requirement corresponding to the higher-level scene.

[0051] The fourth step is to determine the audio enhancement model based on the dedicated model corresponding to the higher-level scenario, in response to the confirmed existence.

[0052] As an example, the aforementioned execution entity can directly determine the dedicated model corresponding to the upper-level scenario as the audio enhancement model.

[0053] In some optional implementations of certain embodiments, the aforementioned execution entity may determine the aforementioned audio enhancement model based on a dedicated model corresponding to the higher-level scenario, including the following steps: The first step is to obtain the audio enhancement training dataset for the aforementioned target scenario. This audio enhancement training data consists of data generated for specific model training based on the various audio enhancement requirements within the target scenario. For example, the various audio enhancement requirements corresponding to the target scenario might include: a first audio enhancement requirement, a second audio enhancement requirement, and a third audio enhancement requirement. The corresponding audio enhancement training data could include: the input audio, the enhanced audio corresponding to the first audio enhancement requirement, the enhanced audio corresponding to the second audio enhancement requirement, and the enhanced audio corresponding to the third audio enhancement requirement. In practice, the audio enhancement training data can be collected from various sources or generated using a dedicated audio enhancement model.

[0054] The second step is to determine the audio enhancement task information corresponding to each of the target scenarios mentioned above. Audio enhancement task information can be the task information corresponding to each audio enhancement task. For example, audio enhancement task information can be a dedicated task identifier corresponding to each audio enhancement task. In practice, there is a one-to-one correspondence between the audio enhancement task information in each audio enhancement task and the audio enhancement requirement in each audio enhancement requirement.

[0055] As an example, information about each audio enhancement task can be determined by querying the audio enhancement requirements associated with the scene.

[0056] The third step is to update the model parameters in the dedicated model corresponding to the above-mentioned audio enhancement task information based on the above-mentioned audio enhancement training dataset, so as to obtain the audio enhancement model.

[0057] As an example, firstly, the output layers corresponding to each audio enhancement task information are removed from the output layers of the dedicated model corresponding to the higher-level scene, resulting in a removed dedicated model. Then, the audio enhancement training dataset is used as the model training dataset for the removed dedicated model, and the model parameters in the removed dedicated model are updated using conventional model training methods to obtain the audio enhancement model.

[0058] In some optional implementations of certain embodiments, after step 203, the steps further include: The first step involves generating at least one set of enhancement training data based on the enhancement effect information corresponding to each of the aforementioned enhanced audios, where the enhancement effect does not meet the corresponding effect conditions. Each enhanced audio has corresponding enhancement effect information, which characterizes the enhancement effect of the enhanced audio. In practice, the enhancement effect information can be in the form of scores or labels. For score-based enhancement effect information, a higher score indicates a better audio enhancement effect and better alignment with the corresponding audio enhancement requirements. In practice, each enhancement effect information can be an audio enhancement effect extracted from user feedback. The effect condition can be that the enhancement effect information corresponding to the enhanced audio meets the requirements of the corresponding audio enhancement needs. For example, for an audio enhancement requirement corresponding to a call task, the effect condition could be that the audio quality after noise removal is higher than the target audio quality. There is a one-to-one correspondence between the enhancement effect information in at least one set of enhancement effect information and the enhancement training data in at least one set of enhancement training data.

[0059] As an example, firstly, at least one enhanced audio corresponding to at least one enhancement effect information is obtained. Then, at least one final enhanced audio is generated for each of the at least one enhanced audio. Each enhanced audio has a unique corresponding final enhanced audio. The final enhanced audio can be the audio after optimizing the enhancement effect of an enhanced audio with poor enhancement effect information. That is, the audio enhancement evaluation effect corresponding to the final enhanced audio is higher than that of the corresponding enhancement effect information. Next, for each final enhanced audio, the audio signal, enhancement effect information, enhanced audio, and final enhanced audio are combined to obtain combined information, which is used as enhancement training data.

[0060] The second step is to store at least one of the above-mentioned enhanced training data into the model database.

[0061] The third step involves setting a ratio between the training data for the target audio enhancement task and the training data for the remaining audio enhancement task sets, in response to the target data volume reaching the target amount stored in the model database. Specifically, the training data volume for the target audio enhancement task is higher than that for the remaining audio enhancement tasks. The set of tasks between the remaining audio enhancement task sets and the target audio enhancement task is the same as the different types of audio enhancement tasks mentioned above (i.e., the remaining audio enhancement task sets are the tasks remaining after removing the target audio enhancement task from the various audio enhancement tasks corresponding to the audio enhancement model). The target audio enhancement task can be the task for which the audio enhancement effect of the audio enhancement model needs to be improved. That is, when the target audio enhancement task is triggered, the audio enhancement model needs to be trained to meet the audio enhancement requirements of the target audio enhancement task. The target data volume can be a measure of whether the amount of training data stored in the model database has reached the threshold for training the model. For example, the target data volume could be 10,000. Each audio enhancement task in the remaining audio enhancement task set is different from the target audio enhancement task. The training data ratio can be the proportion of training data input corresponding to each audio enhancement requirement for training the audio enhancement model. For example, if the target audio enhancement task is task A, and the other audio enhancement tasks include tasks B, C, and D, the training data ratio could be {task A: task B: task C: task D = 4:2:2:2}.

[0062] It should be noted that the proportion of training data can be set on the model training information settings page. This page allows you to configure the information for the audio enhancement model. For example, model information may include: the proportion of training data, model identifier, training epochs, and training method.

[0063] Fourth, based on the aforementioned training data volume ratio, obtain the corresponding augmentation training dataset from the aforementioned model database. Specifically, the amount of augmentation training data for each audio augmentation task in the augmentation training dataset satisfies the training data volume ratio.

[0064] As an example, the aforementioned execution entity can randomly obtain the augmented training dataset corresponding to the data proportion of the audio augmentation task from the aforementioned model database based on the aforementioned training data volume proportion.

[0065] The fifth step involves retraining the audio enhancement model using the aforementioned training dataset to obtain a retrained audio enhancement model for deployment. The specific training method will not be detailed here; it can be any conventional deep learning model training method.

[0066] The various embodiments disclosed above have the following beneficial effects: Through the audio enhancement methods of some embodiments of this disclosure, based on the high speech enhancement performance corresponding to the audio enhancement model, different audio enhancement requirements in the target scenario can be efficiently completed. Specifically, the reason for the low efficiency in fulfilling different audio enhancement requirements is that the allocation of computing resources for audio application scenarios is limited. Different audio enhancement requirements employ different enhancement methods, causing the computational load and parameter count to increase exponentially, leading to a shortage of computing resources. Even the lack of computing resources can result in some audio enhancement requirements not being processed in real time, causing audio output failures. Based on this, the audio enhancement methods of some embodiments of this disclosure first acquire the audio signal to be processed and the reference signal for the target scenario to obtain the audio content to be processed, facilitating subsequent audio content enhancement processing. Then, the audio signal and the reference signal are preprocessed to obtain preprocessed audio feature information, facilitating the extraction of time-domain and / or frequency-domain features, which is beneficial for the subsequent audio enhancement model to learn audio semantic content and for input into the audio enhancement model. Finally, the preprocessed audio feature information is input into the pre-trained audio enhancement model. This allows for efficient processing of various enhanced audio values ​​for different audio enhancement tasks in the target scenario with minimal computational resources. The audio enhancement model supports processing different types of audio enhancement tasks. In summary, using a single audio enhancement model enables efficient implementation of various audio enhancement needs, avoiding the computational complexity and resource constraints caused by requiring multiple implementation methods for different audio enhancement requirements.

[0067] Further reference Figure 3 The diagram illustrates flow 300 of some other embodiments of the audio enhancement method according to this disclosure. The audio enhancement method includes the following steps: Step 301: Obtain the audio signal to be processed and the reference signal for the target scene.

[0068] Step 302: Preprocess the above audio signal and the above reference signal to obtain preprocessed audio feature information.

[0069] In some embodiments, the specific implementation of steps 301-302 and the resulting technical effects can be found in [reference needed]. Figure 2 Steps 201-202 in the corresponding embodiments will not be repeated here.

[0070] Step 303: Input the preprocessed audio feature information into the audio feature extraction layer to obtain audio feature information corresponding to the time domain and frequency domain.

[0071] In some embodiments, the execution entity (e.g. Figure 1The electronic device 101 shown can input the preprocessed audio feature information to the audio feature extraction layer to obtain audio feature information corresponding to the time domain and frequency domain. The audio enhancement model includes: an audio feature extraction layer, a temporal feature extraction layer, a mask generation layer, and an output layer. The audio feature extraction layer can be a network layer that extracts audio feature information. In practice, the audio feature layer can be a network layer based on a convolutional network. For example, the audio feature layer can be a 7-layer concatenated convolutional layer. The audio feature information can be the feature information of the temporal and / or frequency domain semantic content of the preprocessed audio feature information. In practice, the audio feature information can be information in vector form. The temporal feature extraction layer can be a network layer that extracts time-related feature information from the audio feature information. For example, the temporal feature extraction layer can be a GRU (Gated Recurrent Unit) layer. The audio feature extraction layer can only extract local static features, while signals (e.g., speech, music) are dynamic time series and require modeling long-term dependencies (e.g., sentence intonation, musical melody). As a variant of the Recurrent Neural Network (RNN), the GRU controls the transmission of information through update and reset gates. The input to the temporal feature extraction layer is audio feature information, and the output is time-dependent feature information. The mask generation layer can be a network layer that generates a mask matrix with the same dimensions as the matrix corresponding to the preprocessed audio feature information. The mask matrix is ​​information generated by the neural network in tasks such as signal separation, enhancement, and denoising. Essentially, the mask matrix is ​​a matrix of the same size as the input signal (usually a time-frequency domain representation, such as a spectrogram), whose element values ​​are compressed to the [0,1] interval using the sigmoid function, representing the "retention weight" of each time-frequency point for the target signal (0 = complete suppression, 1 = complete retention). The core function of the mask matrix is ​​to "filter" the target component from the mixed signal and suppress interference components through "point-by-point weighting," achieving accurate signal separation or enhancement. The input to the mask generation layer is the output of the temporal feature extraction layer. In practice, the mask generation layer can be composed of at least one fully connected layer. The output layer can be a network layer that multiplies the output of the mask generation layer with the preprocessed audio feature information to obtain at least one enhanced audio corresponding to at least one audio enhancement requirement. In practice, the output layer may include: a multiplication layer, a post-processing layer, and at least one audio output layer. The multiplication layer can be a network layer that performs matrix multiplication on the mask matrix and the preprocessed audio feature information. The post-processing layer can be an inverse STFT (iSTFT) on the matrix output of the multiplication layer to convert the time-frequency domain signal into a time-domain waveform (such as a .wav file). There is a one-to-one correspondence between the audio output layer in the at least one audio output layer and the audio enhancement requirement in the at least one audio enhancement requirement.

[0072] Step 304: Input the above audio feature information into the above time feature extraction layer to obtain time feature information.

[0073] In some embodiments, the execution entity may input the audio feature information into the temporal feature extraction layer to obtain temporal feature information. The temporal feature information may be in vector form, representing time-dependent features.

[0074] Step 305: Input the above time feature information into the above mask generation layer to obtain mask information.

[0075] In some embodiments, the execution entity may input the aforementioned time feature information into the aforementioned mask generation layer to obtain mask information. The mask information may be a mask matrix.

[0076] Step 306: Input the above mask information and the above preprocessed audio feature information into the above output layer to obtain each enhanced audio.

[0077] In some embodiments, the execution entity may input the mask information and the preprocessed audio feature information into the output layer to obtain various enhanced audios.

[0078] In some alternative implementations of certain embodiments, the above audio enhancement model is trained through the following steps: The first step is to acquire the target training data. This target training data includes: audio data, reference data, and at least one audio enhancement data point for the audio data. The target training data can be the training data currently to be used for model training. The audio data can be the audio to be semantically enhanced. The reference data can be the reference signal corresponding to the audio data. The at least one audio enhancement data point can be any data point used as subsequent labels that satisfies the enhancement effect condition (i.e., the condition representing the best enhancement effect). There is a one-to-one correspondence between the audio enhancement data point and at least one audio enhancement requirement.

[0079] The second step is to preprocess the aforementioned target training data to obtain preprocessed data. The specific preprocessing methods will not be elaborated here.

[0080] The third step involves inputting the preprocessed data into the initial audio enhancement model to obtain at least one initial enhanced audio file. The initial audio enhancement model can be an audio enhancement model that has not yet finished training, and the at least one initial enhanced audio file can be audio that has undergone preliminary speech enhancement.

[0081] The fourth step is to determine at least one audio enhancement loss between the at least one audio enhancement data and the at least one initial enhanced audio. There is a one-to-one correspondence between the audio enhancement loss in the at least one audio enhancement loss and the audio enhancement data in the at least one audio enhancement data. The audio enhancement loss characterizes the difference in audio enhancement effect between the audio enhancement data and the corresponding initial enhanced audio. In practice, the higher the loss value of the audio enhancement loss, the greater the difference in the audio enhancement effect.

[0082] As an example, the cross-entropy loss function can be used to determine at least one audio enhancement loss between the at least one audio enhancement data and the at least one initial enhanced audio.

[0083] Fifth step: Based on at least one of the above audio enhancement losses, determine whether the initial audio enhancement model has been trained successfully.

[0084] As an example, at least one audio enhancement loss can be weighted to obtain a weighted loss. Upon determining that the loss change of the weighted loss sequence corresponding to the weighted loss does not converge to the target loss value, it is determined that the initial audio enhancement model training is complete. Each weighted loss in the weighted loss sequence can characterize the loss situation at each training epoch. Upon determining that the loss change of the weighted loss sequence corresponding to the weighted loss converges to the target loss value, it is determined that the initial audio enhancement model training is not complete.

[0085] Step 6: In response to confirming that training is complete, the initial audio enhancement model is identified as the audio enhancement model.

[0086] from Figure 3 It can be seen from this that, with Figure 2 Compared to the description of some corresponding embodiments, Figure 3 The audio enhancement method flow 300 in some corresponding embodiments, based on the audio feature extraction layer, temporal feature extraction layer, mask generation layer and output layer included in the above-mentioned audio enhancement model, can realize the accurate generation of at least one audio enhancement data.

[0087] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an audio enhancement device, which are similar to... Figure 2 Corresponding to the method embodiments shown, this audio enhancement device can be specifically applied to a variety of electronic devices.

[0088] like Figure 4As shown, an audio enhancement device 400 includes an acquisition unit 401, a processing unit 402, and a generation unit 403. The acquisition unit 401 is configured to acquire an audio signal to be processed and a reference signal for a target scene. The processing unit 402 is configured to preprocess the audio signal and the reference signal to obtain preprocessed audio feature information. The generation unit 403 is configured to input the preprocessed audio feature information into a pre-trained audio enhancement model to obtain various enhanced audios corresponding to different audio enhancement tasks in the target scene. The audio enhancement model supports processing different types of audio enhancement tasks.

[0089] In some optional implementations of some embodiments, the processing unit 402 may be further configured to: perform frequency domain transformation on the audio signal and the reference signal to obtain first frequency domain information and second frequency domain information; and concatenate the first frequency domain information and the second frequency domain information to obtain preprocessed audio feature information.

[0090] In some optional implementations of certain embodiments, the audio enhancement model includes: an audio feature extraction layer, a temporal feature extraction layer, a mask generation layer, and an output layer; and the generation unit 403 can be further configured to: input the preprocessed audio feature information into the audio feature extraction layer to obtain audio feature information corresponding to the time domain and frequency domain; input the audio feature information into the temporal feature extraction layer to obtain temporal feature information; input the temporal feature information into the mask generation layer to obtain mask information; and input the mask information and the preprocessed audio feature information into the output layer to obtain various enhanced audios.

[0091] In some optional implementations of certain embodiments, the apparatus 400 further includes: a first determining unit, a second determining unit, a third determining unit, and a fourth determining unit (not shown in the figure). The first determining unit can be configured to: determine whether a dedicated model corresponding to the target scene exists in the model database. The second determining unit can be configured to: determine whether the target scene is a sub-scene in response to determining that it does not exist. The third determining unit can be configured to: determine whether a corresponding dedicated model exists in the parent scene corresponding to the target scene in response to determining that it is a sub-scene. The fourth determining unit can be configured to: determine the audio enhancement model based on the dedicated model corresponding to the parent scene in response to determining that it exists.

[0092] In some optional implementations of some embodiments, the fourth determining unit may be further configured to: acquire an audio enhancement training dataset for the target scenario; determine the audio enhancement task information corresponding to the target scenario; and update the model parameters in the dedicated model corresponding to the upper-level scenario and corresponding to the audio enhancement task information according to the audio enhancement training dataset to obtain an audio enhancement model.

[0093] In some optional implementations of certain embodiments, the apparatus 400 further includes: a data generation unit, a storage unit, a setting unit, a data acquisition unit, and a training unit (not shown in the figure). The data generation unit can be configured to: in response to the existence of enhancement effect information in the enhancement effect information corresponding to each of the enhanced audios where the representation effect does not meet the corresponding effect conditions, generate at least one corresponding enhancement training data based on the at least one unmet enhancement effect information. The storage unit can be configured to: store the at least one enhancement training data in a model database. The setting unit can be configured to: in response to the data volume of the enhancement training data corresponding to the target audio enhancement task stored in the model database reaching a target data volume, set a ratio of the training data volume of the target audio enhancement task to that of the other audio enhancement task sets, wherein the training data volume corresponding to the target audio enhancement task is higher than that corresponding to the other audio enhancement tasks, and the task set between the other audio enhancement task sets and the target audio enhancement task is the same as that of the different types of audio enhancement tasks. The data acquisition unit can be configured to: acquire the corresponding enhancement training dataset from the model database according to the above training data volume ratio. The training unit can be configured to: retrain the audio enhancement model based on the above-mentioned enhanced training dataset to obtain a retrained audio enhancement model for online processing.

[0094] It is understandable that the units described in the audio enhancement device 400 are related to the reference. Figure 2 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the audio enhancement device 400 and the units contained therein, and will not be repeated here.

[0095] The following is for reference. Figure 5 It illustrates electronic devices suitable for implementing some embodiments of this disclosure (e.g., Figure 1 A schematic diagram of the structure of electronic device 101)500. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0096] like Figure 5As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory 502 or a program loaded from a storage device 508 into a random access memory 503. The random access memory 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, the read-only memory 502, and the random access memory 503 are interconnected via a bus 504. An input / output interface 505 is also connected to the bus 504.

[0097] Typically, the following devices can be connected to the input / output interface 505: input devices 506 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 508 including, for example, magnetic tape, hard disk, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 5 Each box shown can represent a device or multiple devices as needed.

[0098] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a read-only memory 502. When the computer program is executed by the processing device 501, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0099] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0100] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0101] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire an audio signal to be processed and a reference signal for a target scene; preprocess the audio signal and the reference signal to obtain preprocessed audio feature information; and input the preprocessed audio feature information into a pre-trained audio enhancement model to obtain various enhanced audios corresponding to different audio enhancement tasks in the target scene, wherein the aforementioned audio enhancement model supports processing different types of audio enhancement tasks.

[0102] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0103] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0104] The units described in some embodiments of this disclosure can be implemented in software or in hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, a processing unit, and a generation unit. The names of these units do not necessarily limit the specific unit; for example, an acquisition unit may also be described as "a unit for acquiring audio signals to be processed and reference signals for a target scene."

[0105] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0106] Some embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements any of the above-described audio enhancement methods.

[0107] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. An audio enhancement method, comprising: Acquire the audio signal to be processed and the reference signal for the target scene; The audio signal and the reference signal are preprocessed to obtain preprocessed audio feature information; The preprocessed audio feature information is input into a pre-trained audio enhancement model to obtain various enhanced audios corresponding to different audio enhancement tasks in the target scene. The audio enhancement model supports the processing of different types of audio enhancement tasks.

2. The method according to claim 1, wherein, The preprocessing of the audio signal and the reference signal to obtain preprocessed audio feature information includes: The audio signal and the reference signal are subjected to frequency domain transformation to obtain first frequency domain information and second frequency domain information; The first frequency domain information and the second frequency domain information are concatenated to obtain preprocessed audio feature information.

3. The method according to claim 1, wherein, The audio enhancement model includes: an audio feature extraction layer, a temporal feature extraction layer, a mask generation layer, and an output layer; and The step of inputting the preprocessed audio feature information into a pre-trained audio enhancement model to obtain various enhanced audios corresponding to different audio enhancement tasks in the target scene includes: The preprocessed audio feature information is input into the audio feature extraction layer to obtain audio feature information corresponding to the time domain and frequency domain. The audio feature information is input into the time feature extraction layer to obtain time feature information; The time feature information is input into the mask generation layer to obtain mask information; The mask information and the preprocessed audio feature information are input into the output layer to obtain various enhanced audios.

4. The method according to claim 1, wherein, The audio enhancement model was trained through the following steps: Acquire target training data, wherein the target training data includes: audio data, reference data, and at least one audio enhancement data for the audio data; The target training data is preprocessed to obtain preprocessed data; The preprocessed data is input into the initial audio enhancement model to obtain at least one initial enhanced audio; Determine at least one audio enhancement loss between the at least one audio enhancement data and the at least one initial enhanced audio; Based on the at least one audio enhancement loss, determine whether the initial audio enhancement model has been trained successfully; In response to the confirmation that training is complete, the initial audio enhancement model is identified as the audio enhancement model.

5. The method according to claim 1, wherein, Before inputting the preprocessed audio feature information into the pre-trained audio enhancement model to obtain the enhanced audio corresponding to different audio enhancement tasks in the target scene, the method further includes... Determine whether a dedicated model corresponding to the target scene exists in the model database; In response to determining that the target scene does not exist, determine whether the target scene is a sub-scene; In response to determining that it is a sub-scene, determine whether there is a corresponding dedicated model for the higher-level scene corresponding to the target scene; In response to the determination of existence, the audio enhancement model is determined based on the dedicated model corresponding to the higher-level scenario.

6. The method according to claim 5, wherein, The process of determining the audio enhancement model based on a dedicated model corresponding to the upper-level scenario includes: Obtain the audio enhancement training dataset for the target scene; Determine the audio enhancement task information corresponding to the target scene; Based on the audio enhancement training dataset, the model parameters in the dedicated model corresponding to the upper-level scene are updated to correspond to the audio enhancement task information, thereby obtaining the audio enhancement model.

7. The method according to claim 1, wherein, The method further includes: In response to the existence of enhancement effect information in each enhancement effect information corresponding to each enhancement audio where the representation effect does not meet the corresponding effect condition, at least one corresponding enhancement training data is generated based on at least one enhancement effect information that does not meet the condition. Store the at least one enhanced training data in the model database; In response to the target data volume of the enhanced training data corresponding to the target audio enhancement task stored in the model database reaching the target data volume, a ratio of the training data volume of the target audio enhancement task to that of the other audio enhancement task sets is set, wherein the training data volume of the target audio enhancement task is higher than that of the other audio enhancement tasks, and the task set between the other audio enhancement task sets and the target audio enhancement task is the same as that of the different types of audio enhancement tasks. Based on the aforementioned proportion of training data, the corresponding augmented training dataset is obtained from the model database; Based on the enhanced training dataset, the audio enhancement model is retrained to obtain a retrained audio enhancement model for online processing.

8. An audio enhancement device, comprising: The acquisition unit is configured to acquire the audio signal to be processed and the reference signal for the target scene; The processing unit is configured to preprocess the audio signal and the reference signal to obtain preprocessed audio feature information; The generation unit is configured to input the preprocessed audio feature information into a pre-trained audio enhancement model to obtain various enhanced audios corresponding to different audio enhancement tasks in the target scene, wherein the audio enhancement model supports the processing of different types of audio enhancement tasks.

9. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images