Audio noise reduction method and device, electronic device and storage medium

Abstract

The application relates to an audio noise reduction method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring multi-modal data, wherein the multi-modal data comprises audio data and associated data of the audio data, and the associated data is used for assisting in understanding the content of the audio data; processing the multi-modal data by using a trained multi-modal audio large model, and outputting the noise type in the audio data; determining a noise reduction strategy corresponding to the noise type according to a preset matching rule; and reducing the noise of the audio data by using the noise reduction strategy. The application can automatically select the optimal noise reduction strategy, thereby improving the noise reduction effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neural networks, and in particular to an audio noise reduction method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In a teleconference, video call or music listening scenario, audio noise reduction is a commonly used technology that can be used to remove or reduce background noise, making target sound (such as human voice, music) clearer and more discernible, and improving the listening experience of users. Current audio noise reduction algorithms mainly include traditional signal processing algorithms and algorithms based on machine learning and deep learning. Traditional signal processing algorithms work well in simple noise environments, but have limited processing capabilities for complex and non-stationary noise. Algorithms based on machine learning and deep learning perform well in complex noise environments, but have high computational complexity and require a large amount of computing resources. Currently, a person needs to manually determine which noise reduction algorithm to use, and the subjectivity of manually determining the noise reduction algorithm is large, which can result in poor noise reduction effect due to inappropriate algorithm selection. SUMMARY

[0003] The present application provides an audio noise reduction method, device, electronic equipment and storage medium to solve the problem of poor noise reduction effect.

[0004] In a first aspect, the present application provides an audio noise reduction method, the method comprising:

[0005] obtaining multi-modal data, wherein the multi-modal data comprises audio data and associated data of the audio data, and the associated data is used to assist in understanding the content of the audio data;

[0006] processing the multi-modal data by a trained multi-modal audio large model to output a noise type in the audio data;

[0007] determining a noise reduction strategy corresponding to the noise type according to a preset matching rule;

[0008] reducing noise in the audio data using the noise reduction strategy.

[0009] Optionally, processing the multi-modal data by the trained multi-modal audio large model to output the noise type in the audio data comprises:

[0010] extracting a plurality of modal features of the multi-modal data by the trained multi-modal audio large model, wherein the plurality of modal features comprises audio features and associated features of the associated data;

[0011] integrating the plurality of modal features into the same vector space using an alignment network;

[0012] In the vector space, an attention mechanism is used to capture the interaction between the audio features and each of the associated features, obtaining a fused multi-modal embedding vector, wherein the multi-modal embedding vector fuses the audio features and all the associated features.

[0013] A fully connected layer is used to extract global semantic features of the multi-modal embedding vector, and a noise type is determined according to the global semantic features, wherein the global semantic features can reflect the overall situation of the audio scene.

[0014] Optionally, capturing the interaction between the audio features and each of the associated features using an attention mechanism to obtain a fused multi-modal embedding vector includes:

[0015] Based on the interaction between the audio features and the associated features at each time, an interaction score is generated;

[0016] The interaction scores of different associated features at the same time are weighted and summed to obtain a context vector, wherein the context vector is used to indicate the audio-related information at each time;

[0017] The audio features and the context vector are combined to generate a fusion feature;

[0018] The multi-modal embedding vector is obtained by integrating the fusion features at all times.

[0019] Optionally, extracting global semantic features of the multi-modal embedding vector using a fully connected layer and determining a noise type according to the global semantic features includes:

[0020] A fully connected layer is used to reduce the dimension of the multi-modal embedding vector to obtain a condensed feature;

[0021] A variety of semantic features in the condensed feature are extracted layer by layer through the fully connected layer to generate a global semantic feature;

[0022] The global semantic feature is matched with a pre-constructed knowledge base to determine the noise type, wherein the knowledge base contains various noise types and their corresponding semantic feature templates.

[0023] Optionally, the associated data includes text and video, and the multi-modal audio large model trained to extract a variety of modal features of the multi-modal data includes:

[0024] The time domain features and frequency domain features of the audio data are extracted by the trained multi-modal audio large model;

[0025] The text features of the text and the visual features of the video are extracted by the trained multi-modal audio large model.

[0026] Optionally, the noise type includes at least one type of noise, and determining the noise reduction strategy corresponding to the noise type according to a preset matching rule includes:

[0027] If the noise type is simple noise, then a noise reduction algorithm in the signal processing algorithm library is determined according to the matching rule;

[0028] If the noise type is complex noise, then a noise reduction algorithm from the deep learning algorithm library is determined according to the matching rules;

[0029] If the noise type includes both simple and complex noise, then a combined noise reduction algorithm from the signal processing algorithm library and the deep learning algorithm library is used according to the matching rules.

[0030] Optionally, the training process for a large multimodal audio model includes:

[0031] Obtain training samples, wherein the training samples include sample audio, noise labels corresponding to the sample audio, and sample association data, wherein the sample association data is used to assist in understanding the content of the sample audio;

[0032] The initial multimodal audio model is trained using the training samples to obtain the output noise classification results;

[0033] If the noise classification result is inconsistent with the noise label, the parameters of the initial multimodal audio large model are adjusted until the noise classification result is consistent with the noise label, thus obtaining the trained multimodal audio large model.

[0034] Secondly, this application provides an audio noise reduction device, the device comprising:

[0035] An acquisition module is used to acquire multimodal data, wherein the multimodal data includes audio data and associated data of the audio data, and the associated data is used to assist in understanding the content of the audio data;

[0036] The processing module is used to process the multimodal audio data by the trained multimodal audio model and output the noise type in the audio data.

[0037] The determination module is used to determine the noise reduction strategy corresponding to the noise type according to the preset matching rules;

[0038] A noise reduction module is used to reduce noise in the audio data using the noise reduction strategy.

[0039] In a third aspect, the present application provides an electronic device, comprising: at least one communication interface; at least one bus connected with the at least one communication interface; at least one processor connected with the at least one bus; and at least one memory connected with the at least one bus.

[0040] In a fourth aspect, the present application further provides a computer storage medium storing computer executable instructions for executing the audio noise reduction method according to any one of the preceding aspects.

[0041] The above technical solution provided by the embodiments of the present application has the following advantages compared with the prior art: the associated data can help better understand the audio content, so that the multi-modal audio large model can combine the associated data and the audio data, accurately identify and classify the noise type in the audio data, and automatically select the optimal noise reduction strategy according to the matching rule, thereby improving the noise reduction effect. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0044] One or more embodiments are illustrated by way of example in the drawings that are for illustrative purposes only, and are not construed to limit the embodiments, and elements having the same reference numerals in the drawings represent similar elements, unless otherwise specified, and the drawings do not constitute a proportional limitation.

[0045] Figure 1 A hardware structure schematic diagram of audio noise reduction provided by the embodiments of the present application;

[0046] Figure 2 A method flowchart of audio noise reduction provided by the embodiments of the present application;

[0047] Figure 3 A structure schematic diagram of an audio noise reduction device provided by the embodiments of the present application;

[0048] Figure 4 A structure schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0049] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described in detail below with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0050] The following disclosure provides many different embodiments or examples for implementing different structures of the present application. In order to simplify the disclosure of the present application, the components and settings of specific examples are described below. Of course, they are only examples and the purpose is not to limit the present application. In addition, reference numerals and / or letters can be repeated in different examples in the present application. Such repetition is for the purpose of simplification and clarity, and does not in itself indicate a relationship between the various embodiments and / or settings discussed.

[0051] In order to solve the problem that the prior art cannot automatically determine the audio noise reduction algorithm, the present application accurately identifies and classifies noise types by combining an audio large model of multi-modal data, and dynamically selects the optimal noise reduction strategy.

[0052] The application scenarios of the noise reduction of the present application include but are not limited to online meetings, public broadcasting, driving environments, film and television shooting, or listening to songs.

[0053] Optionally, in the embodiments of the present application, the above-mentioned audio noise reduction method can be applied in the hardware environment composed of a terminal 101 and a server 103 as shown in Figure 1 As shown in Figure 1 The server 103 is connected with the terminal 101 through a network, which can be used to process the multi-modal data input by the terminal, and a database 105 can be set on the server or independently of the server, which is used to provide data storage services for the server 103. The above-mentioned network includes but is not limited to a wide area network, a metropolitan area network or a local area network, and the terminal 101 includes but is not limited to a PC, a mobile phone, a tablet computer, etc.

[0054] The audio noise reduction method in the embodiments of the present application can be executed by the server 103, and can also be executed by the terminal 101.

[0055] The audio noise reduction method provided by the embodiments of the present application will be described in detail below with the specific implementation manners, taking the server as an example, as shown in Figure 2 The specific steps are as follows:

[0056] Step 201: acquiring multi-modal data, wherein the multi-modal data includes audio data and associated data of the audio data, and the associated data is used to assist in understanding the content of the audio data;

[0057] Step 202: Process the multi-modal data through the trained multi-modal audio large model to output the noise type in the audio data;

[0058] Step 203: Determine the noise reduction strategy corresponding to the noise type according to the preset matching rule;

[0059] Step 204: Use the noise reduction strategy to reduce noise in the audio data.

[0060] In the intelligent audio processing system, the terminal will collect multi-modal data for obvious interference removal, frame, windowing, and other preprocessing operations and upload it to the server. Multi-modal data refers to a dataset that contains multiple different types of information simultaneously, such as audio, text, and video. They collectively describe an event or scene and provide complementary contextual information. Specifically, audio data is the main input source, which can be voice, music, or other environmental sounds recorded by a microphone. Associated data is used to assist in understanding the content of the audio, including but not limited to text descriptions and visual information. Text descriptions are transcriptions of audio into text through automatic speech recognition technology or contextual information provided by users; visual information, such as video frames or images, can provide additional clues about the environment of the audio recording, such as objects in the scene, character actions, etc.

[0061] After the server obtains the above multi-modal data, it processes it using a trained multi-modal audio large model. This model has been trained on a large amount of audio and associated data and has strong semantic understanding and feature extraction capabilities. The functions of the multi-modal audio large model include: integrating audio and associated data, deeply understanding the specific context of the audio, such as distinguishing between human voices, background music, or environmental noise, based on a comprehensive understanding of the content of the audio, accurately identifying and classifying various noise types present in it, such as traffic noise, mechanical noise, and crowd noise.

[0062] Based on the noise type output by the model, the server will refer to the preset matching rules to determine the most appropriate noise reduction strategy. These rules are usually developed by domain experts and take into account the characteristics of different noise types and available noise reduction algorithms. For example, low-frequency noise (such as wind noise, air conditioner operation) prefers adaptive filters; high-frequency sharp noise (such as keyboard typing sound, alarm sound) is more suitable for spectral subtraction; complex mixed noise (such as traffic + crowd noise) recommends a combination algorithm to suppress low-frequency parts first and then process the remaining high-frequency components.

[0063] Finally, the original audio data is processed according to the selected noise reduction strategy. During this process, the server may dynamically adjust parameters to optimize the effect, such as fine-tuning filter settings or updating noise models based on real-time feedback. The server can also perform post-processing operations on the noise-reduced audio signal, such as smoothing, format conversion, etc., and output clear and high-quality audio signals.

[0064] For example, a user records a speech video on a busy city street, and the audio mixes traffic noise, crowd noise and other environmental sounds. The system first obtains the audio and its associated text "Recorded dialogue audio in public place, background with human voice and traffic noise" as multi-modal data. Then the multi-modal audio large model analyzes the audio content and the text content, and identifies that the main noise types are traffic noise and crowd noise. According to the preset rules, the system selects to first use an adaptive filter to suppress low-frequency traffic noise, and then uses a deep learning model to process high-frequency crowd noise. Finally, the output audio has clearer speech and greatly reduced background noise, providing a better listening experience.

[0065] In this application, the associated data can help better understand the audio content, so that the multi-modal audio large model can combine the associated data and the audio data to accurately identify and classify the noise types in the audio data, and automatically select the optimal noise reduction strategy according to the matching rules, thereby improving the noise reduction effect.

[0066] As an optional implementation, in step 202, the multi-modal audio large model is trained to process the multi-modal data, and the noise types in the audio data include the following contents:

[0067] Step S11: Extracting multiple modal features of the multi-modal data by the trained multi-modal audio large model, wherein the multiple modal features include audio features and associated features of the associated data;

[0068] Step S12: Integrating the multiple modal features into the same vector space by using an alignment network;

[0069] Step S13: Capturing the interaction between the audio features and each associated feature in the vector space by using an attention mechanism, to obtain a fused multi-modal embedding vector, wherein the multi-modal embedding vector fuses the audio features and all the associated features;

[0070] Step S14: Extracting global semantic features of the multi-modal embedding vector by using a fully connected layer, and determining the noise type according to the global semantic features, wherein the global semantic features can reflect the overall situation of the audio scene.

[0071] In the intelligent audio processing system, first, rich features need to be extracted from the input multi-modal data, which include not only audio features, but also associated features (such as text features, video features, etc.) related to the audio, to assist in understanding the content of the audio. The features of various modalities are described as follows.

[0072] Audio Features: Extracted from the audio signal, these features reflect the temporal changes and frequency distribution of the audio. They include time-domain features like short-time energy and zero-crossing rate, and frequency-domain features like spectrogram and Mel-frequency cepstral coefficients.

[0073] Text Features: Semantic information extracted from the audio transcription text or user-provided context information. These features help understand the specific context of the audio, especially in tasks like speech recognition and semantic analysis. For example, a user might provide information about the recording location, such as "city square" or "conference room," which helps the system predict possible noise types.

[0074] Visual Features: Visual information about the recording environment extracted from video frames or images. These features provide additional context clues that help accurately determine the audio background. For example, detecting main objects in video frames, such as vehicles, pedestrians, and buildings, helps the system infer possible noise sources.

[0075] Text and visual features are important components of multi-modal data that help the system better understand the audio content and provide important clues for accurately determining noise types. By combining these rich associated information, the system can make more intelligent noise reduction decisions.

[0076] To effectively fuse modality features from different sources in the same framework, the system uses an alignment network to map these features to the same vector space. This step ensures that information from different modalities can be compared and interacted in a unified space, avoiding information loss or confusion that may occur when directly concatenating. It also facilitates meaningful fusion of information from different modalities in the same dimension.

[0077] In the unified vector space, the system uses an attention mechanism to capture the interaction between audio features and each associated feature. The attention mechanism can dynamically focus on key information points, highlighting those associated features that have the most impact on the current audio segment, generating a fused multi-modal embedding vector. This multi-modal embedding vector not only contains the key information of the original audio, but also incorporates important background information from other modalities.

[0078] Finally, the system processes the multi-modal embedding vector through a fully connected layer to extract its global semantic features. These global semantic features can comprehensively reflect the overall situation of the audio scene and be used to determine the types of noise present, such as traffic noise and crowd noise.

[0079] In this application, first, multi-modal audio large models are used to extract various features of audio and associated data, and these features are integrated into the same vector space through an alignment network. Then, in the unified vector space, an attention mechanism is used to capture the interaction between the audio and each associated feature, generating a fused multi-modal embedding vector. Finally, the global semantic features of the embedding vector are extracted through a fully connected layer, so as to accurately determine the noise type. This method not only improves the accuracy of noise reduction, but also enhances the robustness of the system in complex environments.

[0080] As an optional implementation, in step S13, the attention mechanism is used to capture the interaction between the audio features and each associated feature, and the fused multi-modal embedding vector is obtained as follows:

[0081] Step S131: Based on the interaction between the audio features and the associated features at each time, an interaction score is generated;

[0082] Step S132: The interaction scores of different associated features at the same time are weighted and summed to obtain a context vector, wherein the context vector is used to indicate the information associated with the audio at each time;

[0083] Step S133: The audio features and the context vector are combined to generate a fusion feature;

[0084] Step S134: The multi-modal embedding vector is obtained by integrating the fusion features at all times.

[0085] In an intelligent audio processing system, in order to better understand the audio content and improve the noise reduction effect, the system needs to dynamically evaluate the interaction between audio features and associated features (such as text description, visual information) at each time. The specific steps are as follows:

[0086] For each time point, the system calculates the correlation between the audio features and all associated features (such as text features and visual features). These correlations are quantified as interaction scores, which reflect the degree of influence of different associated features on the audio content at that time. For example, if a video frame shows a large number of pedestrians flowing, and there is a crowd noise in the audio at the same time, the interaction score between them will be higher.

[0087] To integrate the information of multiple associated features, the system performs a weighted sum of all interaction scores at the same moment to generate a context vector that synthesizes the important information of all associated features at the current time point, indicating the audio background situation corresponding to the moment. The system calculates attention weights based on the interaction scores to ensure that those associated features considered more important (i.e., higher interaction scores) are given greater weights. The system uses these attention weights to perform a weighted sum of the associated features to form a context vector that not only contains the information of the original associated features but also integrates their interaction relationships with the audio features, providing rich background information for subsequent processing.

[0088] To more comprehensively represent the audio content at each moment, the system combines the original audio features with the generated context vector to form a new fusion feature. The most direct method is to simply concatenate the audio features with the context vector to form a higher-dimensional feature representation.

[0089] Finally, the system integrates all fusion features at all time points to generate a multi-modal embedding vector. This embedding vector synthesizes the multi-modal information of the entire audio segment, providing a basis for further extracting global semantic features. The system can complete the integration process through various methods, such as taking the average, selecting the maximum value, or using a recurrent neural network to capture temporal dependencies. The final multi-modal embedding vector not only contains the key information of the original audio but also integrates important background information from other modalities, enabling it to comprehensively reflect the overall situation of the audio scene.

[0090] In this application, by dynamically evaluating the interaction between audio features and associated features, the system can more accurately capture audio content and its background environment, and then generate a multi-modal embedding vector based on the interaction. The system not only improves audio quality and processing efficiency but also demonstrates how to use data from multiple sources to enhance the intelligent level of audio processing.

[0091] As an optional implementation, in step S14, a fully connected layer is used to extract global semantic features of the multi-modal embedding vector, and the noise type is determined according to the global semantic features, including:

[0092] Step S141: A fully connected layer is used to perform dimensionality reduction processing on the multi-modal embedding vector to obtain condensed features;

[0093] Step S142: Various semantic features in the condensed features are extracted layer by layer through the fully connected layer to generate global semantic features;

[0094] Step S143: The global semantic features are matched with a pre-constructed knowledge base to determine the noise type, wherein the knowledge base contains various noise types and their corresponding semantic feature templates.

[0095] After completing feature extraction, feature alignment, and interaction relationship capture of multi-modal data, the system generates a multi-modal embedding vector that fuses audio features and all associated features. To further optimize processing efficiency and highlight key information, the system next uses a fully connected layer to reduce the dimensionality of this high-dimensional embedding vector.

[0096] The fully connected layer learns to map the high-dimensional multi-modal embedding vector to a lower-dimensional space, generating condensed features. This process not only reduces computational complexity but also enhances the key nature of feature representation by removing redundant information. Despite the reduction in dimensionality, the fully connected layer ensures that condensed features still retain the core information in the original embedding vector, providing a solid foundation for subsequent analysis through carefully designed network structures and parameter adjustments.

[0097] To capture deeper semantic information from condensed features, the system abstracts layer by layer through the fully connected layer, gradually extracting semantic features at different levels. Each layer focuses on a specific type of semantic pattern, ultimately forming a global semantic feature that comprehensively reflects the overall situation of the audio scene. The first layer may focus on basic sound characteristics (such as frequency distribution, pitch variation), while higher layers focus on more complex semantic understanding (such as speech content, environmental description, etc.). This hierarchical processing approach allows the system to gradually refine features that are more representative and explanatory. After multiple layers of processing, the system generates a global semantic feature vector that not only contains key information from the original audio but also integrates important background information from other modalities, thereby comprehensively reflecting the overall situation of the audio scene.

[0098] Finally, the system matches the generated global semantic feature with a pre-constructed knowledge base to determine the noise type that best fits the current audio scene. This knowledge base contains various common noise types and their corresponding semantic feature templates, providing a reliable reference standard for the system. The system calculates the similarity between the global semantic feature and each noise type template in the knowledge base, selecting the closest template as the matching result. For example, if a large number of traffic-related words or visual elements are detected, the system may determine the presence of traffic noise. In some cases, the system may dynamically adjust the noise type judgment in combination with real-time feedback (such as user evaluation or device perception) to ensure accuracy.

[0099] In the present application, the multi-modal embedding vectors are reduced dimensionally by a fully connected layer to generate condensed features, which removes redundant information and ensures the preservation of key features, thereby improving the accuracy of subsequent analysis. Various semantic features in the condensed features are extracted layer by layer to generate global semantic features that comprehensively reflect the overall situation of the audio scene. This enables the system to more accurately capture the audio content and its background environment, reducing false positives. The global semantic features are matched with a pre-constructed knowledge base to ensure that the noise type determination is based on rich templates and empirical data, further improving the accuracy of classification.

[0100] As an optional implementation, in step 203, the noise type includes at least one noise, and the noise reduction strategy corresponding to the noise type is determined according to a preset matching rule, including:

[0101] If the noise type is simple noise, a noise reduction algorithm in the signal processing algorithm library is determined according to the matching rule;

[0102] If the noise type is complex noise, a noise reduction algorithm in the deep learning algorithm library is determined according to the matching rule;

[0103] If the noise type contains simple noise and complex noise, a combined noise reduction algorithm in the signal processing algorithm library and the deep learning algorithm library is used according to the matching rule.

[0104] In an intelligent audio processing system, determining the appropriate noise reduction algorithm is a key step to ensure audio quality. By analyzing the global semantic features and matching the pre-constructed knowledge base, the system can accurately identify the noise type and select the most appropriate noise reduction strategy according to different noise characteristics.

[0105] If the system determines that the noise type is simple noise (such as low-frequency wind noise, air conditioner operation sound, etc.), an appropriate noise reduction algorithm is selected from the signal processing algorithm library according to the preset matching rule. The signal processing algorithm library includes but is not limited to: classic filters such as Wiener filter, Kalman filter, spectral subtraction, and adaptive filter. For simple noise, signal processing algorithms usually have small computational complexity and fast speed, enabling efficient real-time processing on devices with limited resources.

[0106] If the system determines that the noise type is complex noise (such as high-frequency sharp noise, non-stationary noise, or mixed noise), the corresponding noise reduction algorithm is selected from the deep learning algorithm library according to the preset matching rule. The deep learning algorithm library includes but is not limited to: convolutional neural networks, recurrent neural networks and long short-term memory networks, transformer models. Deep learning algorithms can learn and identify complex noise patterns, providing higher noise reduction accuracy, especially in complex environments that are difficult to handle with traditional methods.

[0107] If the system determines that the noise type includes simple noise and complex noise, a combined noise reduction algorithm in the signal processing algorithm library and the deep learning algorithm library is adopted according to a preset matching rule. The simple noise (such as low-frequency traffic noise) is first suppressed using the signal processing algorithm, and then the remaining complex noise (such as crowd noise) is processed through the deep learning algorithm. This phased method can solve different types of problems in a targeted manner. The present application can improve the audio noise reduction effect by selecting the most suitable noise reduction algorithm according to the noise type.

[0108] As an optional implementation, the training process of the multi-modal audio large model includes: obtaining training samples, wherein the training samples include sample audio, noise labels corresponding to the sample audio, and sample associated data for assisting in understanding the content of the sample audio; training the initial multi-modal audio large model using the training samples to obtain output noise classification results; if the noise classification results are inconsistent with the noise labels, using cross-entropy loss function, mean square error loss function and other optimization algorithms to continuously adjust the model parameters to improve the accuracy and generalization ability of the model until the noise classification results are consistent with the noise labels, and a trained multi-modal audio large model is obtained.

[0109] The beneficial effects achieved by the present application are as follows:

[0110] 1. Intelligent noise reduction processing is realized, which can accurately determine the noise type in the audio by combining with the multi-modal audio large model, thereby improving the pertinence and efficiency of noise reduction.

[0111] 2. The utilization of computing resources is optimized, unnecessary complex calculations are avoided, and the real-time performance and resource utilization efficiency of the system are improved.

[0112] 3. The robustness of the system is enhanced, and good noise reduction effect can be maintained even in the case of insufficient training data or complex noise types.

[0113] Based on the same technical concept, the present application provides an audio noise reduction device, as shown in Figure 3 The device includes:

[0114] The acquisition module 301 is configured to acquire multi-modal data, wherein the multi-modal data includes audio data and associated data of the audio data, and the associated data is used to assist in understanding the content of the audio data.

[0115] The processing module 302 is configured to process the multi-modal data by using the trained multi-modal audio large model to output the noise type in the audio data.

[0116] The determination module 303 is configured to determine a noise reduction strategy corresponding to the noise type according to a preset matching rule.

[0117] The noise reduction module 304 is configured to perform noise reduction on the audio data by using a noise reduction strategy.

[0118] Optionally, the processing module 302 is configured to:

[0119] extracting, by the trained multi-modal audio large model, multi-modal feature of the multi-modal data, wherein the multi-modal feature comprises audio feature and association feature of the association data;

[0120] integrating, by using an alignment network, the multi-modal feature into a same vector space;

[0121] capturing, by using an attention mechanism, interaction between the audio feature and each association feature in the vector space to obtain a fused multi-modal embedding vector, wherein the multi-modal embedding vector fuses the audio feature and all the association features;

[0122] extracting, by using a fully connected layer, global semantic feature of the multi-modal embedding vector, and determining the noise type according to the global semantic feature, wherein the global semantic feature can reflect the overall situation of the audio scene.

[0123] Optionally, the processing module 302 is configured to:

[0124] generating an interaction score based on the interaction between the audio feature and the association feature at each time point;

[0125] performing weighted summation on the interaction scores of different association features at the same time point to obtain a context vector, wherein the context vector is used to indicate audio association information at each time point;

[0126] combining the audio feature and the context vector to generate a fused feature;

[0127] obtaining the multi-modal embedding vector by integrating the fused features at all time points.

[0128] Optionally, the processing module 302 is configured to:

[0129] performing dimensionality reduction processing on the multi-modal embedding vector by using a fully connected layer to obtain a condensed feature;

[0130] extracting, by using the fully connected layer, multiple semantic features in the condensed feature layer by layer to generate a global semantic feature;

[0131] matching the global semantic feature with a pre-constructed knowledge base to determine the noise type, wherein the knowledge base contains various noise types and corresponding semantic feature templates of the noise types.

[0132] Optionally, the processing module 302 is configured to:

[0133] extracting, by the trained multi-modal audio large model, time domain feature and frequency domain feature of the audio data;

[0134] The completed multi-modal audio large model extracts text features of the text and extracts visual features of the video.

[0135] Optionally, the noise type includes at least one noise, and the determination module 303 is configured to:

[0136] If the noise type is simple noise, a noise reduction algorithm in the signal processing algorithm library is determined according to the matching rule;

[0137] If the noise type is complex noise, a noise reduction algorithm in the deep learning algorithm library is determined according to the matching rule;

[0138] If the noise type contains simple noise and complex noise, a combined noise reduction algorithm in the signal processing algorithm library and the deep learning algorithm library is used according to the matching rule.

[0139] Optionally, the training process of the multi-modal audio large model includes:

[0140] Obtaining a training sample, wherein the training sample includes a sample audio, a noise label corresponding to the sample audio, and sample associated data for assisting in understanding the content of the sample audio;

[0141] Using the training sample to train the initial multi-modal audio large model to obtain an output noise classification result;

[0142] If the noise classification result is inconsistent with the noise label, adjusting the parameters of the initial multi-modal audio large model until the noise classification result is consistent with the noise label, and obtaining the trained multi-modal audio large model.

[0143] As shown in Figure 4 The embodiment of the present application provides an electronic device, which comprises a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402 and the memory 403 complete mutual communication through the communication bus 404.

[0144] The memory 403 is used to store a computer program.

[0145] In an embodiment of the present application, the processor 401 is used to execute the program stored in the memory 403, and realizes the audio noise reduction method provided by any one of the preceding method embodiments.

[0146] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by the processor to realize the steps of the audio noise reduction method provided by any one of the preceding method embodiments.

[0147] The apparatus embodiments described above are only illustrative, and units described as separate units can or can not be physically separate, and units shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, and the computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the embodiments or some parts of the embodiments.

[0149] It should be understood that the terms used herein are for the purpose of describing particular example embodiments only and are not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises", "comprising", "includes", "including" and "has" are inclusive and therefore specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order in which they are described, unless specifically identified as an order dependent step. It is also to be understood that additional or alternative steps can be employed.

[0150] The above description is merely illustrative of the application and should not be taken as limiting. Numerous modifications and variations underlying the general principles of the applications can be made by those of ordinary skill in the art without departing from the spirit or scope of the application. Therefore, the application is not to be limited to the embodiments described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An audio noise reduction method, characterized in that, The method includes: Acquire multimodal data, wherein the multimodal data includes audio data and associated data of the audio data, and the associated data is used to assist in understanding the content of the audio data; The multimodal audio model, once trained, is used to process the multimodal data and output the noise type in the audio data. The noise reduction strategy corresponding to the noise type is determined according to the preset matching rules; The noise reduction strategy described above is used to reduce noise in the audio data; The multimodal audio model, after training, processes the multimodal data and outputs the noise types in the audio data, including: The trained multimodal audio model extracts multiple modal features from the multimodal data, including audio features and association features of the associated data. An alignment network is used to integrate the various modal features into a single vector space; In the vector space, an attention mechanism is used to capture the interaction between the audio features and each associated feature to obtain a fused multimodal embedding vector, wherein the multimodal embedding vector fuses the audio features and all associated features; A fully connected layer is used to extract global semantic features from the multimodal embedding vector, and the noise type is determined based on the global semantic features, wherein the global semantic features can reflect the overall situation of the audio scene; The method employs an attention mechanism to capture the interaction between the audio features and each associated feature, resulting in a fused multimodal embedding vector comprising: An interaction score is generated based on the interaction relationship between the audio features and the associated features at each time point; The interaction scores of different association features at the same time are weighted and summed to obtain a context vector, which is used to indicate the audio association information at each time. The audio features are combined with the context vector to generate fused features; The multimodal embedding vector is obtained by integrating the fusion features at all time points.

2. The method according to claim 1, characterized in that, The global semantic features of the multimodal embedding vector are extracted using a fully connected layer, and the noise type is determined based on the global semantic features, including: The multimodal embedding vectors are reduced in dimensionality using a fully connected layer to obtain condensed features; The fully connected layer extracts multiple semantic features from the condensed features layer by layer to generate global semantic features. The global semantic features are matched with a pre-built knowledge base to determine the noise type, wherein the knowledge base contains various noise types and their corresponding semantic feature templates.

3. The method according to claim 1, characterized in that, The associated data includes text and video. Multiple modal features of the multimodal data are extracted using a trained multimodal audio model, including: The time-domain and frequency-domain features of the audio data are extracted using a trained multimodal audio model. The trained multimodal audio model extracts textual features from the text and visual features from the video.

4. The method according to claim 1, characterized in that, The noise type includes at least one type of noise, and the noise reduction strategy corresponding to the noise type is determined according to a preset matching rule, including: If the noise type is simple noise, then a noise reduction algorithm in the signal processing algorithm library is determined according to the matching rule; If the noise type is complex noise, then a noise reduction algorithm from the deep learning algorithm library is determined according to the matching rules; If the noise type includes both simple and complex noise, then a combined noise reduction algorithm from the signal processing algorithm library and the deep learning algorithm library is used according to the matching rules.

5. The method according to claim 1, characterized in that, The training process of the multimodal audio large model includes: Obtain training samples, wherein the training samples include sample audio, noise labels corresponding to the sample audio, and sample association data, wherein the sample association data is used to assist in understanding the content of the sample audio; The initial multimodal audio model is trained using the training samples to obtain the output noise classification results; If the noise classification result is inconsistent with the noise label, the parameters of the initial multimodal audio large model are adjusted until the noise classification result is consistent with the noise label, thus obtaining the trained multimodal audio large model.

6. An audio noise reduction device, characterized in that, The device includes: An acquisition module is used to acquire multimodal data, wherein the multimodal data includes audio data and associated data of the audio data, and the associated data is used to assist in understanding the content of the audio data; The processing module is used to process the multimodal audio data by the trained multimodal audio model and output the noise type in the audio data. The determination module is used to determine the noise reduction strategy corresponding to the noise type according to the preset matching rules; A noise reduction module is used to reduce noise in the audio data using the noise reduction strategy. The processing module is used for: The trained multimodal audio model extracts multiple modal features from the multimodal data, including audio features and association features of the associated data. An alignment network is used to integrate the various modal features into a single vector space; In the vector space, an attention mechanism is used to capture the interaction between the audio features and each associated feature to obtain a fused multimodal embedding vector, wherein the multimodal embedding vector fuses the audio features and all associated features; A fully connected layer is used to extract global semantic features from the multimodal embedding vector, and the noise type is determined based on the global semantic features, wherein the global semantic features can reflect the overall situation of the audio scene; Specifically, the processing module is used for: An interaction score is generated based on the interaction relationship between the audio features and the associated features at each time point; The interaction scores of different association features at the same time are weighted and summed to obtain a context vector, which is used to indicate the audio association information at each time. The audio features are combined with the context vector to generate fused features; The multimodal embedding vector is obtained by integrating the fusion features at all time points.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-5.

Images