An audio enhancement method and device based on a large acoustic model, equipment and medium

Abstract

The application provides an audio enhancement method and device based on a hearing large model, equipment and a medium. The method comprises: acquiring an acoustic feature sequence, a hidden feature sequence, a speech recognition text sequence, a scene category and a sound event set corresponding to an original audio signal, and an enhancement task prompt text corresponding to the original audio signal; inputting the acoustic feature sequence, the speech recognition text sequence, the scene category and the enhancement task prompt text into the hearing large model to obtain a processing type allocation result corresponding to the original audio signal; inputting the hidden feature sequence, the speech recognition text sequence, the scene category, the sound event set, the enhancement task prompt text and the processing type allocation result into the hearing large model to obtain an enhanced audio signal. Through the application, key acoustic events can be accurately distinguished and retained based on semantic understanding, and fine audio enhancement for business perception is realized.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing, and more specifically, to an audio enhancement method, apparatus, device, and medium based on a large auditory model. Background Technology

[0002] In fields such as voice communication, remote work, intelligent recording, telemedicine, intelligent customer service, and industrial operation and maintenance, audio is often collected in complex acoustic environments, containing target speech, multi-speaker speech, steady-state background noise, sudden environmental sounds, and sound events with specific business implications. The basic idea of ​​traditional scene-aware speech enhancement technology is to first determine the environment and then select a processing strategy. That is, the system first identifies the environment or noise type of the audio through acoustic features, and then selects a noise reduction model, noise reduction parameters, or audio processing mode that matches the environment from several pre-designed processing modes.

[0003] However, such methods typically treat all ambient sounds other than the target speech as objects to be suppressed. This is fixed and detached from the business context, resulting in a lack of refined semantic discrimination in the audio enhancement process. While indiscriminately filtering out noise, it is very easy to mistakenly remove various key acoustic events with business value, thereby reducing the business integrity and scene authenticity of the enhanced audio. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an audio enhancement method, apparatus, device and medium based on a large auditory model, which can accurately distinguish and retain key acoustic events based on semantic understanding, and realize refined audio enhancement for business perception.

[0005] In a first aspect, embodiments of this application provide an audio enhancement method for a large auditory model, the method comprising: Acquire the acoustic feature sequence, hidden feature sequence, speech recognition text sequence, scene category, and sound event set corresponding to the original audio signal, as well as the enhanced task prompt text corresponding to the original audio signal; the hidden feature sequence fuses the acoustic features and contextual semantic information corresponding to the original audio signal. The acoustic feature sequence, the speech recognition text sequence, the scene category, and the enhanced task prompt text are input into the allocation module of the auditory big data model to obtain the processing type allocation result corresponding to the original audio signal; the processing type allocation result includes the target processing type assigned to each sound source in the original audio signal. The hidden feature sequence, the speech recognition text sequence, the scene category, the sound event set, the enhancement task prompt text, and the processing type allocation result are input into the strategy generation module of the auditory big model to obtain the enhancement strategy corresponding to the original audio signal; the enhancement strategy includes processing parameters corresponding to each processing type; The original audio signal, the hidden feature sequence, the processing type assignment result, and the enhancement strategy are input into the conditional audio enhancement decoder of the auditory big model to obtain the enhanced audio signal.

[0006] In one possible implementation, the target processing type corresponding to any sound source is determined according to the following steps: Based on the acoustic feature representation of the target audio segment where the sound source is located in the acoustic feature sequence, the speech recognition text sequence, the scene category, and the enhanced task prompt text, the allocation probability of each processing type is determined for the sound source; The processing type with the highest allocation probability is determined as the target processing type corresponding to the sound source.

[0007] In one possible implementation, the step of inputting the hidden feature sequence, the speech recognition text sequence, the scene category, the sound event set, the enhancement task prompt text, and the processing type allocation result into the policy generation module of the auditory big model to obtain the enhancement policy corresponding to the original audio signal includes: Substituting the hidden feature sequence, the speech recognition text sequence, the scene category, the sound event set, the enhancement task prompt text, and the processing type allocation result into the following formula, the enhancement strategy corresponding to the original audio signal is obtained; ; ; in, To enhance the strategy; For learnable mapping functions, For the learnable weight matrix of the policy generation module, The global hidden vector is obtained by converging the hidden feature sequences. This is the convergence feature vector corresponding to the set of sound events; The aggregation representation corresponding to the processing type; Representation for scene categories; This represents the corresponding text sequence for speech recognition. To enhance the representation of task prompt text; The learnable bias vector of the policy generation module; To enhance the intensity of speech; Enhance the strength of effective non-voice events; Background noise suppression intensity; This represents the reverberation suppression intensity.

[0008] In one possible implementation, the step of inputting the original audio signal, the hidden feature sequence, the processing type assignment result, and the enhancement strategy into the conditional audio enhancement decoder of the large auditory model to obtain the enhanced audio signal includes: Based on the original audio signal and the hidden feature sequence, a mask vector corresponding to each sound source is generated; each mask vector contains the proportion of the corresponding sound source at each video position; Based on the processing type allocation result and the enhancement strategy, determine the processing weight corresponding to each sound source; The original audio signal is processed according to the mask vector and processing weight corresponding to each sound source to obtain the enhanced audio signal.

[0009] In one possible implementation, determining the processing weights corresponding to each sound source based on the processing type allocation result and the enhancement strategy includes: Substituting the processing type allocation result and the enhancement strategy into the following formula, the processing weights corresponding to each sound source are obtained: ; ; in, The processing weights corresponding to sound source s; The speech enhancement intensity in the enhancement strategy; Strengthen the effective non-voice events in the enhancement strategy; This is the first adjustment coefficient; This is the second adjustment coefficient; The background noise suppression intensity in the enhancement strategy; Assign the target processing type to the sound source s in the processing type assignment result; This is the j-th processing type.

[0010] In one possible implementation, the step of enhancing the original audio signal based on the mask vectors and processing weights corresponding to each sound source to obtain the enhanced audio signal includes: Substitute the mask vectors corresponding to each sound source, the processing weights corresponding to each sound source, and the original audio signal into the following formula to obtain the enhanced audio signal; ; in, The enhanced audio signal; The processing weights corresponding to the kth speech source; This is the mask vector corresponding to the kth target speaker; For the first The processing weights corresponding to each non-speech event sound source; For the third party other than the target speaker The mask vectors corresponding to each sound source; This is the original audio signal.

[0011] In one possible implementation, the large auditory model is trained according to the following steps: Acquire audio sample signals and the corresponding audio tag dataset; the audio tag dataset includes processing type assignment result tags; With the goal of minimizing the loss function value, the auditory big model is trained based on the audio sample signals and their corresponding audio label datasets. The loss function includes a semantic preservation constraint loss term. The semantic preservation constraint loss term assigns labels according to the processing type, constraining the auditory big model to retain the key acoustic events that need to be enhanced and suppress the interfering acoustic events that need to be suppressed during the audio enhancement process, thereby maintaining the scene semantic information of the enhanced audio.

[0012] Secondly, embodiments of this application also provide an audio enhancement device based on a large auditory model, the device comprising: The acquisition module is used to acquire the acoustic feature sequence, hidden feature sequence, speech recognition text sequence, scene category and sound event set corresponding to the original audio signal, as well as the enhanced task prompt text corresponding to the original audio signal; the hidden feature sequence integrates the acoustic features and contextual semantic information corresponding to the original audio signal. The input module is used to input the acoustic feature sequence, the speech recognition text sequence, the scene category, and the enhanced task prompt text into the allocation module of the auditory big model to obtain the processing type allocation result corresponding to the original audio signal; the processing type allocation result includes the target processing type assigned to each sound source in the original audio signal. The input module is further configured to input the hidden feature sequence, the speech recognition text sequence, the scene category, the sound event set, the enhancement task prompt text, and the processing type allocation result into the strategy generation module of the auditory big model to obtain the enhancement strategy corresponding to the original audio signal; the enhancement strategy includes processing parameters corresponding to each processing type; The input module is further configured to input the original audio signal, the hidden feature sequence, the processing type allocation result, and the enhancement strategy into the conditional audio enhancement decoder of the auditory big model to obtain the enhanced audio signal.

[0013] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the audio enhancement method based on a large auditory model as described in any of the first aspects.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the audio enhancement method based on a large auditory model as described in any of the first aspects.

[0015] This application provides an audio enhancement method, apparatus, device, and medium based on an auditory big model. The method includes: acquiring an acoustic feature sequence, a hidden feature sequence, a speech recognition text sequence, a scene category, and a set of sound events corresponding to an original audio signal, as well as enhancement task prompt text corresponding to the original audio signal; inputting the acoustic feature sequence, the speech recognition text sequence, the scene category, and the enhancement task prompt text into an auditory big model to obtain a processing type allocation result corresponding to the original audio signal; and inputting the hidden feature sequence, the speech recognition text sequence, the scene category, the set of sound events, the enhancement task prompt text, and the processing type allocation result into the auditory big model to obtain an enhanced audio signal. This application enables precise differentiation and retention of key acoustic events based on semantic understanding, achieving refined audio enhancement for business-aware purposes. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart of an audio enhancement method based on a large auditory model provided in an embodiment of this application is shown; Figure 2 This document illustrates an audio enhancement flowchart provided in an embodiment of this application. Figure 3 A schematic diagram of the structure of an audio enhancement device based on a large auditory model provided in an embodiment of this application is shown; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0019] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0020] To enable those skilled in the art to utilize the content of this application, and in conjunction with the specific application scenario of "audio signal processing," the following implementation methods are provided. For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application is primarily described within the "audio signal processing field," it should be understood that this is merely an exemplary embodiment.

[0021] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0022] The following is a detailed description of an audio enhancement method based on a large auditory model provided in the embodiments of this application.

[0023] Reference Figure 1 The diagram shown is a flowchart illustrating an audio enhancement method based on a large auditory model according to an embodiment of this application. The exemplary steps of this embodiment are described below: S101. Obtain the acoustic feature sequence, hidden feature sequence, speech recognition text sequence, scene category and sound event set corresponding to the original audio signal, as well as the enhanced task prompt text corresponding to the original audio signal.

[0024] In this embodiment, the original audio signal and its corresponding enhanced task prompt text are obtained; the original audio signal is input into the audio encoder of the auditory big data model to obtain an acoustic feature sequence; the acoustic feature sequence is input into the context modeling module in the joint understanding network of the auditory big data model to obtain a hidden feature sequence; the acoustic feature sequence is input into the extraction module in the joint understanding network of the auditory big data model to obtain the speech recognition text sequence, scene category and sound event set corresponding to the original audio signal.

[0025] The augmentation task prompt text is used to explain the current augmentation task objective to the auditory big model. It can be a fixed general prompt that instructs the model to identify speech content, ambient sound and scene in the audio and generate selective augmentation strategies accordingly. It can also be a customized prompt that is tied to specific business, such as prompting the model that the current audio is a remote consultation recording and that the sound related to the consultation needs to be retained.

[0026] Specifically, the original audio signal is input into the audio encoder of the large auditory model to obtain the acoustic feature sequence: the audio encoder, as the front end of the large auditory model, encodes the original audio signal, extracts multi-scale acoustic features, and constitutes the acoustic feature sequence A.

[0027] The original audio signal can be a time-domain signal, or a spectrum obtained by time-frequency transformation, or a neural audio tag or other equivalent acoustic representation input into the audio encoder of the auditory large model.

[0028] Specifically, the acoustic feature sequence is input into the context modeling module in the joint understanding network of the JueDa model to obtain the hidden feature sequence: the context modeling module performs context modeling on the acoustic feature sequence to obtain the hidden feature sequence H; the hidden feature sequence integrates the acoustic features corresponding to the original audio signal and the semantic information of the context.

[0029] Hidden features are internal feature representations formed by the model when processing audio tags. They contain both acoustic information of the corresponding audio segment and semantic information obtained by combining context. This sequence of hidden features serves as input to the subsequent conditional audio enhancement decoder, allowing audio understanding and audio enhancement to share the same set of internal acoustic representations.

[0030] Specifically: The acoustic feature sequence is input into the extraction module of the joint understanding network of the auditory big data model to obtain the speech recognition text sequence, scene category, and sound event set corresponding to the original audio signal. (1) Speech recognition text sequence: Speech-recognized text is not only a human-readable transcription result, but its semantics will also serve as an important basis for processing type assignment and enhancement strategy generation.

[0031] For example, when a patient describes coughing or wheezing in the speech recognition text, and the sound event detection results show both coughing and wheezing events, the corresponding sound event can be elevated to an object that needs to be retained or enhanced; when an inspector describes abnormal machine noises, and abnormal friction or impact sounds are detected in the audio, the abnormal sound event can be given a higher priority for retention or enhancement.

[0032] It should be noted that the speech recognition text is only used to provide semantic context, and whether the sound event actually exists is still determined by the environmental sound event detection results.

[0033] (2) Scene category: The scenarios can be categorized as remote consultations, customer service quality inspections, conference calls, industrial inspections, law enforcement recordings, in-vehicle calls, or classroom recordings, etc. The auditory big data model outputs scenario results based on global audio semantics, which are used to constrain subsequent processing type assignments and enhancement strategy generation.

[0034] (3) Set of sound events: The sound event set contains all the sound sources in the original audio signal and their occurrence times within the original audio signal. Sound sources can include speakers, air conditioner noise, keyboard clicks, voices in the corridor, coughs, wheezing, sirens, impact sounds, mechanical friction sounds, wind noise, electrical noise, etc. Sound event detection can be achieved using multi-label recognition or event detection methods with time localization. The sound event set provides a time localization basis for subsequent event-based differentiated enhancement.

[0035] S102. Input the acoustic feature sequence, speech recognition text sequence, scene category, and enhanced task prompt text into the allocation module of the auditory big model to obtain the processing type allocation result corresponding to the original audio signal.

[0036] In this embodiment, not all non-speech sounds are assumed to be noise. Instead, a processing type is dynamically assigned to each type of sound source based on the semantics of the scene. The processing type is obtained by classifying different enhancement processing methods (such as preservation, enhancement, suppression, weakening, etc.). The processing type assignment result includes the target processing type assigned to each sound source in the original audio signal.

[0037] For example, the processing type set can be defined as: ; in, This indicates enhancement, meaning actively increasing the energy and clarity of the sound source. This indicates preservation, meaning that the original characteristics of the sound source should be maintained as much as possible without being destroyed; This indicates neutral treatment, meaning that the sound source is not significantly enhanced or suppressed; This indicates weakening, meaning that the energy of the sound source is moderately reduced but not completely eliminated; This indicates suppression, meaning eliminating the sound as much as possible. The above five processing types are arranged as follows: to The corresponding processing intensities decrease sequentially.

[0038] In addition, the set of processing types is not limited to the five categories of enhancement, retention, suppression, weakening and neutral mentioned above. It can also be expanded or replaced with more semantically targeted processing types according to specific business needs, such as foreground sound, background sound, evidence sound, medical clue sound, security warning sound, privacy sensitive sound, diagnostic sound and equipment failure sound, etc.

[0039] Specifically, the allocation module determines the target processing type corresponding to any sound source according to the following steps: Step 1: Based on the acoustic feature representation of the target audio segment where the sound source is located in the acoustic feature sequence, the speech recognition text sequence, the scene category, and the enhanced task prompt text, determine the allocation probability of each processing type for the sound source.

[0040] In the embodiments of this application, for each sound source The auditory big data model extracts the acoustic feature representation of the target audio segment corresponding to the sound source. Scene category correspondence representation Speech recognition text sequence correspondence representation Corresponding to the enhanced task prompt text Calculate the allocation probability of each processing type for this sound source:

[0041] Among them, sound source It can be the voice of a specific speaker or the sound of an environmental source; This indicates vector concatenation; The learnable weight matrix for the allocation module; Learnable bias vectors for the allocation module; As the sound source In processing type collections The probability distribution vector on, its th Each component Represented as a sound source Allocation processing type The probability of assignment; softmax is the activation function.

[0042] Step 2: Determine the processing type with the highest allocation probability as the target processing type corresponding to the sound source.

[0043] In the embodiments of this application, the sound source The final target processing type Choose the one with the highest probability:

[0044] This yields the results of the processing type allocation for all sound sources in this inference. This includes both the target processing type for each speaker's speech and the target processing type for each environmental sound source. Since the processing type assignment is conditional on the scene, speech recognition semantics, and task, the same sound source will receive different processing types in different contexts. For example, in a remote consultation scenario, patient speech is assigned enhancement, coughing and wheezing are assigned preservation due to their diagnostic value, air conditioning noise and hallway noise unrelated to diagnosis are assigned suppression, and keyboard noise is assigned weakening. In an industrial inspection scenario, abnormal equipment friction noise is assigned enhancement or preservation because it is a fault clue, staff speech is assigned preservation, wind noise is assigned suppression, and ordinary background noise is assigned weakening.

[0045] S103. Input the hidden feature sequence, speech recognition text sequence, scene category, sound event set, enhanced task prompt text, and the processing type allocation result into the strategy generation module of the auditory big model to obtain the enhancement strategy corresponding to the original audio signal.

[0046] In this embodiment, the enhancement strategy can provide global, adjustable control parameters for the subsequent conditional audio enhancement decoder, including processing parameters corresponding to each processing type, which can be represented as follows: .

[0047] in, To enhance the intensity of speech; Enhance the strength of effective non-voice events; Background noise suppression intensity; This represents the reverberation suppression intensity.

[0048] Here, speech enhancement intensity refers to the speech enhancement intensity of the target speaker (the core effective speaker that needs to be preserved and clearly restored, and is the subject of protection in the audio enhancement task), effective non-speech event enhancement intensity refers to the enhancement intensity of other sound sources that are effective non-target speakers, background noise suppression intensity refers to the suppression intensity of background noise, and reverberation suppression intensity refers to the suppression intensity of room reverberation, echo tail, and spatial reflection residual signals in the audio.

[0049] Furthermore, enhancement strategies can be in the form of continuous vectors or discrete control tag sequences, such as control tag expressions based on scenarios like patient consultation, preserving cough, preserving wheezing, suppressing air conditioning noise, and enhancing patient speech. In addition, they can be expressed in various forms, such as structured text, as long as they can represent the control intent of how to process different sounds.

[0050] Specifically, the hidden feature sequence, the speech recognition text sequence, the scene category, the sound event set, the enhancement task prompt text, and the processing type allocation result are substituted into the following formula to obtain the enhancement strategy corresponding to the original audio signal; ; in, To enhance the strategy; For learnable mapping functions, For the learnable weight matrix of the policy generation module, The global hidden vector is obtained by converging the hidden feature sequences. This is the convergence feature vector corresponding to the set of sound events; The aggregation representation corresponding to the processing type; Representation for scene categories; This represents the corresponding text sequence for speech recognition. To enhance the representation of task prompt text; Learnable bias vectors of the policy generation module.

[0051] Here, the processing mode of existing technologies is determined solely by the scenario: ;in, For processing mode, This is a fixed mapping from scene category to processing mode. However, the enhancement strategy in this application's embodiments is determined by multiple semantic paths: ;in, Indicates by scene category Speech recognition text sequence Sound event set Processing type allocation results and enhance task prompt text To Enhancement Strategy The mapping. That is, the enhancement strategy depends simultaneously on the scene, speech recognition content, sound events, processing type, and task objective. It is this from arrive The extension enables the embodiments of this application to generate enhancement strategies that are far more refined and better suited to business needs than fixed mappings.

[0052] S104. Input the original audio signal, hidden feature sequence, processing type assignment result and enhancement strategy into the conditional audio enhancement decoder of the auditory big model to obtain the enhanced audio signal.

[0053] In the embodiments of this application, reference is made to Figure 2 The diagram shown is an audio enhancement flowchart provided in an embodiment of this application. Specifically, the original audio signal, the hidden feature sequence, the processing type allocation result, and the enhancement strategy are input into the conditional audio enhancement decoder of the auditory large model according to the following steps to obtain the enhanced audio signal: S201. Generate mask vectors for each sound source based on the original audio signal and the hidden feature sequence.

[0054] In this embodiment, each mask vector contains the proportion of the corresponding sound source at each video location. This is to determine the proportion of each sound source in the original audio signal. The corresponding time-frequency region in the original audio signal is used by the conditional audio enhancement decoder. and hidden feature sequences Generate mask vectors for each sound source:

[0055] in, This represents the mask generation network. The number of target speakers in the original audio signal. The number of sound sources other than the target speaker in the original audio signal; This is the mask vector corresponding to the kth target speaker; For the third party other than the target speaker The mask vectors corresponding to each sound source.

[0056] S202. Based on the processing type allocation results and enhancement strategy, determine the processing weight corresponding to each sound source.

[0057] In this embodiment of the application, when the processing type set includes five categories: enhancement, preservation, suppression, weakening, and neutral, the processing type allocation result and the enhancement strategy are substituted into the following formula to obtain the processing weight corresponding to each sound source: ; ; in, The processing weights corresponding to sound source s; The speech enhancement intensity in the enhancement strategy; Strengthen the effective non-voice events in the enhancement strategy; This is the first adjustment coefficient; This is the second adjustment coefficient; The background noise suppression intensity in the enhancement strategy; Assign the target processing type to the sound source s in the processing type assignment result; This is the j-th processing type.

[0058] here, Sound sources marked as enhanced have a processing weight greater than 1, and their enhancement magnitude is determined by... or The weight of a sound source marked as preserved is 1; the weight of a sound source marked as neutral, weakened, or suppressed varies with the input. Increases and decreases. Due to The attenuation of a neutral sound source is less than that of a weakened sound source; when When the sound source is marked as suppressed, the weight is 0.

[0059] S203. Based on the mask vector and processing weight corresponding to each sound source, the original audio signal is processed to obtain the enhanced audio signal.

[0060] In this embodiment of the application, the mask vector corresponding to each sound source, the processing weight corresponding to each sound source, and the original audio signal are substituted into the following formula to obtain the enhanced audio signal. ; in, The enhanced audio signal; The processing weights corresponding to the kth speech source; This is the mask vector corresponding to the kth target speaker; For the first The processing weights corresponding to each non-speech event sound source; For the third party other than the target speaker The mask vectors corresponding to each sound source; This is the original audio signal.

[0061] Here, this form supports both enhancing only a single target speech and simultaneously enhancing the speech of multiple target speakers. The reverberation suppression intensity is not included in the above source weight calculation, but is used to control the dereverberation processing intensity in the enhancement decoder. In specific implementations, to avoid amplitude anomalies caused by overlapping multiple source masks or enhancement weights greater than 1, the weighted total mask can be normalized, amplitude-limited, and energy-constrained. The above formula clearly reflects the core of the embodiments of this application, namely, the enhancement result is no longer simply retaining the target speech while eliminating all other sounds, but rather assigning different retention weights to different sounds according to the scene semantics, thereby achieving differentiated and interpretable selective enhancement.

[0062] It should be noted that the enhancement decoder can be implemented using various network structures, including but not limited to frequency domain masking networks, encoder-decoder networks, temporal convolutional decoupling networks, deep audio decoupling models, diffusion models, neural vocoders, audio codec tag generation models, and attention-based audio decoders. Alternatively, the decoder can directly output the enhanced waveform without explicitly calculating the frequency domain mask and its corresponding inverse transform.

[0063] In addition, this application embodiment also provides a training process for a large auditory model, the implementation steps of which are as follows: Step 1: Obtain the audio sample signal and the corresponding audio tag dataset.

[0064] In this embodiment of the application, an audio sample signal containing multiple annotations is constructed. The audio label dataset labeled with each audio sample signal contains target-enhanced audio labels. Speech recognition text sequence labels Scene category tags Sound event collection tags Processing type assignment result label It can also include enhanced strategy tags. Or a weak supervision strategy label.

[0065] Step 2: With the minimum loss function value as the optimization objective, train the auditory big model based on the audio sample signals and their corresponding audio label datasets; the loss function includes a semantic preservation constraint loss term; the semantic preservation constraint loss term assigns labels according to the processing type, constraining the auditory big model to retain the key acoustic events that need to be enhanced and suppress the interfering acoustic events that need to be suppressed during the audio enhancement process, thus maintaining the scene semantic information of the enhanced audio.

[0066] In this embodiment, since multiple tasks are jointly performed in the large auditory model, the overall loss function is composed of a weighted sum of the losses of each sub-task:

[0067] in, For the large auditory model, the loss function is... For speech recognition text loss terms; The loss term is classified according to the scene. Loss items for sound event detection; Assign a loss term to the processing type; For audio enhancement loss terms; To retain the constraint loss term; These are the weighting coefficients corresponding to the text loss term in speech recognition; The weight coefficients corresponding to the scene classification loss term; The weighting coefficients corresponding to the sound event detection loss term; Assign weight coefficients to the loss terms based on the processing type; The weighting coefficients for the audio enhancement loss term; To retain the weighting coefficients corresponding to the constraint loss terms; to These are the weighting coefficients for each loss item, which can be adjusted according to the task's focus.

[0068] For example, one specific way to determine each loss item is as follows: (1) Speech recognition text loss term Used to measure the speech recognition text and speech recognition text sequence labels predicted by the large auditory model. The difference between them can be expressed using label-by-label cross-entropy:

[0069] in, For the auditory large model based on audio sample signals Corresponding acoustic feature sequence Task prompts Given the preceding predicted text tags before the t-th text tag, predict the probability of the current t-th text tag; To indicate the length of the text; Let t be the t-th text tag in the speech recognition text sequence labeling. The loss function primarily utilizes the speech content in the audio and ASR annotations. Sure.

[0070] (2) Scene classification loss term Used to measure the scene predicted by the large auditory model and the scene category label. The difference between them can be expressed using cross-entropy:

[0071] in, For the auditory large model based on audio sample signals Corresponding global hidden vector Predicted scene categories The probability of this loss is calculated using global acoustic and semantic features of the entire audio segment, as well as scene category labels. Sure.

[0072] (3) Loss item for sound event detection Used to measure the sound events predicted by the large auditory model and their timing in the audio, along with the sound event set labels. The differences between them. Suppose the audio is divided into... There are [number] time units, totaling [number] time units. Sound-like events, and They represent the first In the time unit, the th The labeled values ​​and predicted probabilities of sound-like events can be obtained using multi-label binary cross-entropy:

[0073] (4) Assign loss items based on processing type Used to measure the processing type assignment results of the auditory large model prediction versus the processing type assignment results. The differences between them. Suppose there are a total of [number] elements in an audio clip. The first sound source, in the processing type assignment result, is the... The actual processing type of each sound source is: Then cross-entropy can be used:

[0074] in, For the large auditory model, the sound source Predicted as the actual processing type The probability of.

[0075] (5) Audio enhancement loss term Audio sample signals used to measure the output of a large auditory model Corresponding enhanced audio , with target enhanced audio tags The differences between them can be compared simultaneously in terms of their time-domain waveforms and spectra.

[0076] The first item represents waveform difference, and the second item represents spectral difference. and For the corresponding weights, It is a short-time Fourier transform. This loss primarily utilizes the enhanced audio from the model output. and target enhanced audio Sure.

[0077] (6) Semantic Preservation Constraint Loss Term This loss is used to determine whether the enhanced audio retains key acoustic events that should be enhanced or preserved, and reduces acoustic events that should be suppressed. This loss primarily utilizes the enhanced audio... Sound event collection tags Assign result labels to processing type Confirmed, details are as follows.

[0078] This is a key training constraint that distinguishes this application's embodiments from traditional speech enhancement: the enhanced audio must not only be clear but also retain the key sound events marked as preserved or enhanced, while effectively reducing the sound events marked as suppressed. To this end, the enhanced audio can be... The confidence levels of sound events for each sound event are obtained by feeding sound event recognition heads, frozen teacher event recognizers, or independent sound event detectors into the same large auditory model. .

[0079] Define the set of event indices that are marked as enhanced or retained:

[0080] And define the set of indices of the events to be suppressed that are marked as suppressed:

[0081] Based on this definition, semantic preservation constraint loss is adopted, using hinge loss:

[0082] in, The processing type in the result label assigned to the processing type belongs to the set of acoustic events that need to be preserved or enhanced; The processing type in the result label assigned to the processing type belongs to the set of acoustic events that need to be suppressed; To preset the retention threshold, when based on enhanced audio The confidence level corresponding to the predicted i-th sound event A penalty is applied when the value falls below the preset retention threshold, prompting the model to retain key acoustic events; To preset the suppression threshold, when based on enhanced audio The confidence level corresponding to the predicted i-th sound event A penalty is applied when the noise level exceeds the preset suppression threshold, prompting the auditory model to suppress interfering sounds. It is this loss term that enables the auditory model to no longer only pursue clear human voices when enhancing audio, but to constrain the preservation of scene-meaning sounds and the suppression of irrelevant noise at the optimization objective level, thereby overcoming the shortcomings of existing technologies in erroneously deleting key sounds from the training mechanism.

[0083] Based on the same inventive concept, this application also provides an audio enhancement device based on an auditory large model, which corresponds to the audio enhancement method based on an auditory large model. Since the principle of the device in this application is similar to the audio enhancement method based on an auditory large model described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0084] Reference Figure 3The diagram shown is a schematic of an audio enhancement device based on a large auditory model according to an embodiment of this application. The audio enhancement device based on a large auditory model includes: The acquisition module 301 is used to acquire the acoustic feature sequence, hidden feature sequence, speech recognition text sequence, scene category and sound event set corresponding to the original audio signal, as well as the enhanced task prompt text corresponding to the original audio signal; the hidden feature sequence integrates the acoustic features and contextual semantic information corresponding to the original audio signal. Input module 302 is used to input the acoustic feature sequence, the speech recognition text sequence, the scene category, and the enhanced task prompt text into the allocation module of the auditory big model to obtain the processing type allocation result corresponding to the original audio signal; the processing type allocation result includes the target processing type assigned to each sound source in the original audio signal; The input module 302 is further configured to input the hidden feature sequence, the speech recognition text sequence, the scene category, the sound event set, the enhancement task prompt text, and the processing type allocation result into the strategy generation module of the auditory big model to obtain the enhancement strategy corresponding to the original audio signal; the enhancement strategy includes processing parameters corresponding to each processing type; The input module 302 is further configured to input the original audio signal, the hidden feature sequence, the processing type allocation result, and the enhancement strategy into the conditional audio enhancement decoder of the auditory big model to obtain the enhanced audio signal.

[0085] In one possible implementation, the input module 302 is specifically configured to determine the target processing type corresponding to any sound source according to the following steps: determining the allocation probability of each processing type for the sound source based on the acoustic feature representation of the target audio segment where the sound source is located in the acoustic feature sequence, the speech recognition text sequence, the scene category and the enhanced task prompt text; and determining the processing type with the highest allocation probability as the target processing type corresponding to the sound source.

[0086] In one possible implementation, the input module 302 is specifically used to substitute the hidden feature sequence, the speech recognition text sequence, the scene category, the sound event set, the enhancement task prompt text, and the processing type allocation result into the following formula to obtain the enhancement strategy corresponding to the original audio signal; ; ; in, To enhance the strategy; For learnable mapping functions, For the learnable weight matrix of the policy generation module, The global hidden vector is obtained by converging the hidden feature sequences. This is the convergence feature vector corresponding to the set of sound events; The aggregation representation corresponding to the processing type; Representation for scene categories; This represents the corresponding text sequence for speech recognition. To enhance the representation of task prompt text; The learnable bias vector of the policy generation module; To enhance the intensity of speech; Enhance the strength of effective non-voice events; Background noise suppression intensity; This represents the reverberation suppression intensity.

[0087] In one possible implementation, the input module 302 is specifically configured to generate mask vectors corresponding to each sound source based on the original audio signal and the hidden feature sequence; each mask vector contains the proportion of the corresponding sound source at each video position; determine the processing weights corresponding to each sound source based on the processing type allocation result and the enhancement strategy; and process the original audio signal based on the mask vectors and processing weights corresponding to each sound source to obtain the enhanced audio signal.

[0088] In one possible implementation, the input module 302 is specifically used to substitute the processing type allocation result and the enhancement strategy into the following formula to obtain the processing weights corresponding to each sound source: ; ; in, The processing weights corresponding to sound source s; The speech enhancement intensity in the enhancement strategy; Strengthen the effective non-voice events in the enhancement strategy; This is the first adjustment coefficient; This is the second adjustment coefficient; The background noise suppression intensity in the enhancement strategy; Assign the target processing type to the sound source s in the processing type assignment result; This is the j-th processing type.

[0089] In one possible implementation, the input module 302 is specifically used to substitute the mask vectors corresponding to each sound source, the processing weights corresponding to each sound source, and the original audio signal into the following formula to obtain the enhanced audio signal. ; in, The enhanced audio signal; The processing weights corresponding to the kth speech source; This is the mask vector corresponding to the kth target speaker; For the first The processing weights corresponding to each non-speech event sound source; For the third party other than the target speaker The mask vectors corresponding to each sound source; This is the original audio signal.

[0090] In one possible implementation, the device further includes a training module 303, which is used to train the auditory big model according to the following steps: acquiring audio sample signals and audio label datasets corresponding to the audio sample signals; the audio label datasets include processing type assignment result labels; training the auditory big model based on the audio sample signals and their corresponding audio label datasets with the minimum loss function value as the optimization objective; the loss function includes a semantic preservation constraint loss term; the semantic preservation constraint loss term assigns labels according to the processing type, constraining the auditory big model to retain key acoustic events that need to be enhanced and suppress interfering acoustic events that need to be suppressed during the audio enhancement process, thereby maintaining the scene semantic information of the enhanced audio.

[0091] like Figure 4 As shown in the embodiment of this application, an electronic device 400 includes a processor 401, a memory 402, and a bus. The memory 402 stores machine-readable instructions executable by the processor 401. When the electronic device is running, the processor 401 communicates with the memory 402 via the bus, and the processor 401 executes the machine-readable instructions to perform the steps of the method based on the auditory big model described above.

[0092] Specifically, the memory 402 and processor 401 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 401 runs the computer program stored in the memory 402, it can execute the above-mentioned method based on the auditory big model.

[0093] Corresponding to the above-described method based on the auditory big model, this application embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described method based on the auditory big model.

[0094] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0095] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0096] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0097] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the auditory big model-based method described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, external hard drives, ROM, RAM, magnetic disks, or optical disks.

[0098] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An audio enhancement method based on a large auditory model, characterized in that, The method includes: Acquire the acoustic feature sequence, hidden feature sequence, speech recognition text sequence, scene category, and sound event set corresponding to the original audio signal, as well as the enhanced task prompt text corresponding to the original audio signal; the hidden feature sequence fuses the acoustic features and contextual semantic information corresponding to the original audio signal. The acoustic feature sequence, the speech recognition text sequence, the scene category, and the enhanced task prompt text are input into the allocation module of the auditory big data model to obtain the processing type allocation result corresponding to the original audio signal; the processing type allocation result includes the target processing type assigned to each sound source in the original audio signal. The hidden feature sequence, the speech recognition text sequence, the scene category, the sound event set, the enhancement task prompt text, and the processing type allocation result are input into the strategy generation module of the auditory big model to obtain the enhancement strategy corresponding to the original audio signal; the enhancement strategy includes processing parameters corresponding to each processing type; The original audio signal, the hidden feature sequence, the processing type assignment result, and the enhancement strategy are input into the conditional audio enhancement decoder of the auditory big model to obtain the enhanced audio signal.

2. The audio enhancement method based on a large auditory model according to claim 1, characterized in that, Determine the target processing type for any sound source using the following steps: Based on the acoustic feature representation of the target audio segment where the sound source is located in the acoustic feature sequence, the speech recognition text sequence, the scene category, and the enhanced task prompt text, the allocation probability of each processing type is determined for the sound source; The processing type with the highest allocation probability is determined as the target processing type corresponding to the sound source.

3. The audio enhancement method based on a large auditory model according to claim 1, characterized in that, The step of inputting the hidden feature sequence, the speech recognition text sequence, the scene category, the sound event set, the enhancement task prompt text, and the processing type allocation result into the policy generation module of the auditory big model to obtain the enhancement policy corresponding to the original audio signal includes: Substituting the hidden feature sequence, the speech recognition text sequence, the scene category, the sound event set, the enhancement task prompt text, and the processing type allocation result into the following formula, the enhancement strategy corresponding to the original audio signal is obtained; ； ； in, To enhance the strategy; For learnable mapping functions, For the learnable weight matrix of the policy generation module, The global hidden vector is obtained by converging the hidden feature sequences. This is the convergence feature vector corresponding to the set of sound events; The aggregation representation corresponding to the processing type; Representation for scene categories; This represents the corresponding text sequence for speech recognition. To enhance the representation of task prompt text; The learnable bias vector of the policy generation module; To enhance the intensity of speech; Enhance the strength of effective non-voice events; Background noise suppression intensity; This represents the reverberation suppression intensity.

4. The audio enhancement method based on a large auditory model according to claim 1, characterized in that, The process of inputting the original audio signal, the hidden feature sequence, the processing type assignment result, and the enhancement strategy into the conditional audio enhancement decoder of the large auditory model to obtain the enhanced audio signal includes: Based on the original audio signal and the hidden feature sequence, a mask vector corresponding to each sound source is generated; each mask vector contains the proportion of the corresponding sound source at each video position; Based on the processing type allocation result and the enhancement strategy, determine the processing weight corresponding to each sound source; The original audio signal is processed according to the mask vector and processing weight corresponding to each sound source to obtain the enhanced audio signal.

5. The audio enhancement method based on a large auditory model according to claim 4, characterized in that, The step of determining the processing weights corresponding to each sound source based on the processing type allocation result and the enhancement strategy includes: Substituting the processing type allocation result and the enhancement strategy into the following formula, the processing weights corresponding to each sound source are obtained: ； ； in, The processing weights corresponding to sound source s; The speech enhancement intensity in the enhancement strategy; Strengthen the effective non-voice events in the enhancement strategy; This is the first adjustment coefficient; This is the second adjustment coefficient; The background noise suppression intensity in the enhancement strategy; Assign the target processing type to the sound source s in the processing type assignment result; This is the j-th processing type.

6. The audio enhancement method based on a large auditory model according to claim 4, characterized in that, The step of enhancing the original audio signal based on the mask vector and processing weights corresponding to each sound source to obtain the enhanced audio signal includes: Substitute the mask vectors corresponding to each sound source, the processing weights corresponding to each sound source, and the original audio signal into the following formula to obtain the enhanced audio signal; ； in, The enhanced audio signal; The processing weights corresponding to the kth speech source; This is the mask vector corresponding to the kth target speaker; For the first The processing weights corresponding to each non-speech event sound source; For the third party other than the target speaker The mask vectors corresponding to each sound source; This is the original audio signal.

7. The audio enhancement method based on a large auditory model according to claim 1, characterized in that, The large auditory model is trained according to the following steps: Acquire audio sample signals and the corresponding audio tag dataset; the audio tag dataset includes processing type assignment result tags; With the goal of minimizing the loss function value, the auditory large model is trained based on the audio sample signals and their corresponding audio label datasets. The loss function includes a semantic preservation constraint loss term; the semantic preservation constraint loss term is labeled according to the processing type, which constrains the auditory big model to retain the key acoustic events that need to be enhanced and suppress the interfering acoustic events that need to be suppressed during the audio enhancement process, thus maintaining the scene semantic information of the enhanced audio.

8. An audio enhancement device based on a large auditory model, characterized in that, The device includes: The acquisition module is used to acquire the acoustic feature sequence, hidden feature sequence, speech recognition text sequence, scene category and sound event set corresponding to the original audio signal, as well as the enhanced task prompt text corresponding to the original audio signal; the hidden feature sequence integrates the acoustic features and contextual semantic information corresponding to the original audio signal. The input module is used to input the acoustic feature sequence, the speech recognition text sequence, the scene category, and the enhanced task prompt text into the allocation module of the auditory big model to obtain the processing type allocation result corresponding to the original audio signal; the processing type allocation result includes the target processing type assigned to each sound source in the original audio signal. The input module is further configured to input the hidden feature sequence, the speech recognition text sequence, the scene category, the sound event set, the enhancement task prompt text, and the processing type allocation result into the strategy generation module of the auditory big model to obtain the enhancement strategy corresponding to the original audio signal; the enhancement strategy includes processing parameters corresponding to each processing type; The input module is further configured to input the original audio signal, the hidden feature sequence, the processing type allocation result, and the enhancement strategy into the conditional audio enhancement decoder of the auditory big model to obtain the enhanced audio signal.

9. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the audio enhancement method based on a large auditory model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the audio enhancement method based on a large auditory model as described in any one of claims 1 to 7.

Images