Audio noise reduction method, system and device based on AI model and storage medium

Abstract

The invention discloses an audio noise reduction method, system and device based on an AI model and a storage medium, and relates to the technical field of voice noise reduction, and the method comprises the steps: receiving a live video stream, and segmenting the live video stream into image frame data and audio stream data; the image frame data and the audio stream data are input to a preset scene recognition model, the recognition result of the current scene is obtained, and the preset scene recognition model comprises a character recognition sub-model, a voice recognition sub-model and a classification output module; and determining and executing a current noise reduction strategy according to the identification result to obtain an audio output signal, the current noise reduction strategy being a human voice enhancement strategy, an ambient sound enhancement strategy or a fusion enhancement strategy. According to the invention, scene recognition is carried out in a multi-modal fusion mode, different noise reduction strategies are adaptively executed according to the scene recognition result, noise optimization can be carried out for different scenes, and the audio output effect is improved.

Description

Technical Field

[0001] The present application relates to the field of speech noise reduction technology, and in particular to an audio noise reduction method, system, device and storage medium based on an AI model. Background Art

[0002] Noise is a significant factor affecting the quality of voice interaction. Traditional noise reduction algorithms are particularly effective for stationary noise. However, real-world noise is often non-stationary, meaning its characteristics change over time. Consequently, when outputting audio and video in changing scenarios, such as live streaming, traditional noise reduction algorithms struggle to adaptively adjust noise reduction parameters in these diverse environments, especially those with unknown noise levels. This results in poor audio signal quality. Summary of the Invention

[0003] The main purpose of this application is to provide an audio noise reduction method, system, device and storage medium based on an AI model, aiming to solve the technical problem that traditional noise reduction algorithms are difficult to perform differentiated noise reduction processing in different scenarios, thereby affecting the output effect of audio signals.

[0004] To achieve the above objectives, the present application proposes an audio noise reduction method based on an AI model, which includes:

[0005] Receive a live video stream, and split the live video stream into image frame data and audio stream data;

[0006] Inputting the image frame data and the audio stream data into a preset scene recognition model to obtain a recognition result of the current scene, wherein the preset scene recognition model includes a person recognition sub-model, a speech recognition sub-model, and a classification output module, wherein the person recognition sub-model is used to determine scene person information based on the image frame data, the speech recognition sub-model is used to determine scene audio information based on the audio stream data, and the classification output module is used to determine the scene category of the current scene based on the scene person information and the scene audio information;

[0007] A current noise reduction strategy is determined and executed according to the recognition result to obtain an audio output signal, wherein the current noise reduction strategy is: a human voice enhancement strategy, an ambient sound enhancement strategy, or a fusion enhancement strategy.

[0008] In one embodiment, the step of inputting the image frame data and the audio stream data into a preset scene recognition model to obtain a recognition result of the current scene includes:

[0009] Extracting features from the image frame data using the character recognition sub-model to obtain visual features, and obtaining scene character information based on the visual features;

[0010] Extracting features from the audio stream data using the speech recognition sub-model to obtain sound features, and obtaining scene audio information based on the sound features;

[0011] The classification output module receives the scene character information and the scene audio information, and determines the scene category of the current scene according to the scene character information and the scene audio information. The scene categories include: character scene, scenery scene and fusion scene.

[0012] In one embodiment, the step of obtaining scene character information based on the visual features includes:

[0013] Determine whether there is a subject person in the current scene based on the visual features, and obtain a subject person determination result;

[0014] The scene character information is obtained according to the main character judgment result, and the scene character information is: the main character exists or does not exist.

[0015] In one embodiment, the step of obtaining scene audio information according to the sound features includes:

[0016] Determine whether there is a human voice in the current scene based on the sound characteristics, and obtain a human voice determination result;

[0017] The scene audio information is obtained according to the character voice judgment result, and the scene audio information is: the presence of the character voice or the absence of the character voice.

[0018] In one embodiment, the step of determining the scene category of the current scene based on the scene character information and the scene audio information includes:

[0019] When the scene character information indicates that a subject character exists and the scene audio information indicates that a character voice exists, obtaining facial features of the subject character according to the visual features;

[0020] Determining whether the facial features are consistent with the voice features;

[0021] If there is consistency, determining that the current scene is a character scene;

[0022] If there is no consistency, the current scene is determined to be a fusion scene.

[0023] In one embodiment, the step of determining the scene category of the current scene based on the scene character information and the scene audio information further includes:

[0024] When the scene character information indicates that there is no main character and the scene audio information indicates that there is a character voice, if the volume of the character voice is greater than a preset volume value, determining that the current scene is a fusion scene;

[0025] When the scene audio information shows that there is no human voice, it is determined that the current scene is a landscape scene.

[0026] In one embodiment, the step of determining and executing a current noise reduction strategy based on the recognition result to obtain an audio output signal includes:

[0027] When the current scene is a character scene, executing a human voice enhancement strategy to perform noise reduction processing on the audio stream data to obtain an audio output signal, wherein the human voice enhancement strategy is a strategy for enhancing the voice of the main character while reducing the ambient sound;

[0028] When the current scene is a landscape scene, executing an ambient sound enhancement strategy to perform noise reduction processing on the audio stream data to obtain an audio output signal, wherein the ambient sound enhancement strategy is a strategy for enhancing only the ambient sound;

[0029] When the current scene is a fusion scene, a fusion enhancement strategy is executed to perform noise reduction processing on the audio stream data to obtain an audio output signal. The fusion enhancement strategy is a strategy for simultaneously enhancing character voices and ambient sounds.

[0030] In addition, to achieve the above purpose, this application also proposes an audio noise reduction system based on an AI model, the system comprising:

[0031] A video segmentation module is used to receive a live video stream and split the live video stream into image frame data and audio stream data;

[0032] a scene recognition module, configured to input the image frame data and the audio stream data into a preset scene recognition model to obtain a recognition result of the current scene, wherein the preset scene recognition model includes a person recognition sub-model, a speech recognition sub-model, and a classification output module, wherein the person recognition sub-model is configured to determine scene person information based on the image frame data, the speech recognition sub-model is configured to determine scene audio information based on the audio stream data, and the classification output module is configured to determine the scene category of the current scene based on the scene person information and the scene audio information;

[0033] The strategy matching module is used to determine and execute the current noise reduction strategy according to the recognition result to obtain an audio output signal, where the current noise reduction strategy is: human voice enhancement strategy, ambient sound enhancement strategy or fusion enhancement strategy.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes an audio noise reduction device based on an AI model, the device comprising: a memory, a processor, and an audio noise reduction program stored on the memory and runnable on the processor, the audio noise reduction program being configured to implement the steps of the audio noise reduction method based on the AI model as described above.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, on which an audio noise reduction program is stored. When the audio noise reduction program is executed by a processor, the steps of the audio noise reduction method based on the AI model as described above are implemented.

[0036] The present application discloses an audio noise reduction method based on an AI model, which receives a live video stream and divides the live video stream into image frame data and audio stream data; inputs the image frame data and the audio stream data into a preset scene recognition model to obtain a recognition result of the current scene, wherein the preset scene recognition model includes a person recognition sub-model, a speech recognition sub-model, and a classification output module, wherein the person recognition sub-model is used to determine scene person information based on the image frame data, the speech recognition sub-model is used to determine scene audio information based on the audio stream data, and the classification output module is used to determine the scene category of the current scene based on the scene person information and the scene audio information; determines and executes the current noise reduction strategy based on the recognition result to obtain an audio output signal, wherein the current noise reduction strategy is: a human voice enhancement strategy, an ambient sound enhancement strategy, or a fusion enhancement strategy. In the present application, by combining the image information and audio information in the video to adopt a multimodal fusion method to perform scene recognition in the video, the robustness of scene recognition in complex or changing environments can be improved. Different noise reduction strategies can be dynamically executed based on the scene recognition results, such as voice enhancement, scene sound enhancement or fusion enhancement strategy. The noise reduction strategy can be adaptively adjusted according to the current scene, which is conducive to improving the audio output effect in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1 This is a flowchart of the first embodiment of the audio noise reduction method based on the AI model of this application;

[0040] Figure 2 This is a flowchart of the second embodiment of the audio noise reduction method based on the AI model of this application;

[0041] Figure 3 This is a flowchart of the third embodiment of the audio noise reduction method based on the AI model of this application;

[0042] Figure 4 This is a schematic diagram of the module structure of the first embodiment of the audio noise reduction system based on the AI model of this application;

[0043] Figure 5 This is a structural diagram of the audio noise reduction device based on the AI model in this application.

[0044] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0046] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0047] The present invention provides an audio noise reduction method based on an AI model. Figure 1 , Figure 1 This is a flow chart of the first embodiment of the audio noise reduction method based on the AI model of this application. In this embodiment, the method includes: steps S10 to S30:

[0048] Step S10: receiving a live video stream, and dividing the live video stream into image frame data and audio stream data.

[0049] It should be noted that the method of this embodiment can be used in scenarios such as live video broadcasting or video conferencing. The execution subject of the method of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a mobile phone, tablet, speaker, sound column and audio processor, etc. It can also be other electronic devices with the same or similar functions that can implement the audio noise reduction method based on the AI model of this application. Here, the audio noise reduction device based on the AI model (hereinafter referred to as the "noise reduction device") is used as an example to specifically describe the various embodiments of this application.

[0050] It is understandable that the noise reduction device can obtain the live video stream pushed by the live broadcast end, and divide the live video stream into several video segments based on the similarity of video frames, each video segment includes several similar continuous frames.

[0051] It should be understood that for each video segment, decoding technology can be used to separate image frame data and audio stream data from the original data stream, thereby facilitating the improvement of the accuracy of subsequent scene recognition through multimodal data fusion.

[0052] Step S20: Input the image frame data and the audio stream data into a preset scene recognition model to obtain a recognition result of the current scene, wherein the preset scene recognition model includes a character recognition sub-model, a speech recognition sub-model and a classification output module, wherein the character recognition sub-model is used to determine scene character information based on the image frame data, the speech recognition sub-model is used to determine scene audio information based on the audio stream data, and the classification output module is used to determine the scene category of the current scene based on the scene character information and the scene audio information.

[0053] It should be understood that video frame data and audio stream data are input into a pre-set scene recognition model. This scene recognition model may include multiple sub-models, such as person recognition and speech recognition, as well as a classification output module. This multimodal fusion approach can improve the performance of machine learning models by establishing connections between different layers of the model, allowing information fused at deeper levels to be used at shallower levels, thereby improving recognition accuracy and robustness.

[0054] It should be noted that the person recognition sub-model and speech recognition sub-model can be trained using a neural network model using a scene image dataset and an audio dataset, respectively. The person recognition sub-model can be used to analyze the number of people in a scene within a video frame, including their number, location, and even their expressions. The speech recognition sub-model can use sound spectrum analysis to distinguish between human voices and ambient sounds in audio, as well as the content and characteristics of the sounds.

[0055] It should also be noted that the preset scene recognition model can also be trained using a neural network model using scene image and audio datasets. The classification output module can derive fused features based on the scene character information obtained by the above-mentioned character recognition sub-model and the scene audio information obtained by the above-mentioned speech recognition sub-model. The fused features can then be used to ultimately classify and output the corresponding scene category, such as a concert, speech, or outdoor activity.

[0056] The classification output module may also store mapping relationships between combinations of different scene character information and scene audio information and scene categories, and thus may directly map corresponding scene categories based on the scene character information and scene audio information received in real time.

[0057] It can be understood that the classification output module integrates image information and audio information to determine the category of the current scene, thereby reducing the "semantic gap", that is, the difference between high-level semantic representation and low-level feature representation, and thus improving the accuracy of scene recognition.

[0058] Step S30: determining and executing a current noise reduction strategy according to the recognition result to obtain an audio output signal, wherein the current noise reduction strategy is: a human voice enhancement strategy, an ambient sound enhancement strategy, or a fusion enhancement strategy.

[0059] It should be understood that the noise reduction device can automatically match the most suitable noise reduction strategy based on the recognized scene category. This adaptive method can optimize the processing based on the characteristics of the current scene and improve the noise reduction effect.

[0060] It should be noted that noise reduction strategies may include: voice enhancement strategy, ambient sound enhancement strategy, or fusion enhancement strategy. Each noise reduction strategy is applicable to different scene categories. The specific scene categories applicable to different noise reduction strategies can be pre-defined so that the current noise reduction strategy corresponding to the current scene can be directly determined based on the recognition results.

[0061] For example, the voice enhancement strategy can be applied in scenarios where the human voice needs to be highlighted (such as speeches and interviews), to enhance speech clarity while reducing background noise.

[0062] The scene sound enhancement strategy can be applied to videos of concerts or natural scenery, focusing on improving the ambient sound effects and creating an immersive experience.

[0063] The fusion enhancement strategy can be applied to complex scenarios and comprehensively optimizes the audio quality by combining the enhancement of human voice and ambient sound.

[0064] In a specific implementation, the noise reduction device matches the corresponding noise reduction strategy according to the identified scene category, and then executes the noise reduction strategy to obtain the audio stream data after noise reduction processing, and outputs the audio stream data as the audio of the live video stream.

[0065] This embodiment uses a multimodal fusion approach to identify scenes in videos by combining image and audio information. This improves the robustness of scene recognition in complex or changing environments. Based on the scene recognition results, different noise reduction strategies are dynamically implemented, such as voice enhancement, scene sound enhancement, or fusion enhancement. This adaptively adjusts the noise reduction strategy based on the current scene, improving audio output quality in different scenarios.

[0066] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 2 , Figure 2 This is a flow chart of the second embodiment of the audio noise reduction method based on the AI model of this application.

[0067] In this embodiment, in order to illustrate how to implement the scene recognition process based on the preset scene recognition model, step S20 specifically includes: steps S201 to S203:

[0068] Step S201: extracting features from the image frame data using the character recognition sub-model to obtain visual features, and obtaining scene character information based on the visual features.

[0069] It should be understood that the person recognition sub-model can be constructed based on a convolutional neural network (CNN) or other visual recognition architecture. The person recognition sub-model can extract visual features from the image frame data, such as edges, textures, shapes, and spatial relationships. Using the extracted visual features, the person recognition sub-model can identify information about people in the image, including their presence, location, actions, and possible emotional state.

[0070] It should also be noted that in order to determine whether the current scene is a scene that needs to highlight the human voice (character scene), it is possible to judge whether there is a main character in the current scene based on visual features, obtain the main character judgment result, and then determine the scene character information based on the main character judgment result. The scene character information can be whether there is a main character or not.

[0071] Exemplarily, a person recognition sub-model constructed based on CNN may include: 3 convolutional layers, 3 pooling layers, 3 ReLU activation layers, 1 flattening layer, 1 fully connected layer and 1 output layer.

[0072] The process of extracting features from the image frame data using the person recognition sub-model may be:

[0073] First, the image frame data is input into the CNN model. The shape of an input image in the image frame data is 224*224*3, which respectively represent the width, height and number of color channels of the image;

[0074] Next, in convolution layer 1, 32 3x3 convolution kernels are used to perform a convolution operation on the input image to extract low-level features such as edges and textures of the image. The shape of the feature map obtained after convolution is (224, 224, 32).

[0075] In the ReLU activation layer 1, the ReLU activation function is applied to the output of the convolution layer 1 to increase the nonlinear expression ability of the model, and the output of the ReLU activation layer 1 is batch normalized to accelerate the training process and improve the stability of the model.

[0076] In pooling layer 1, a 2x2 maximum pooling operation is used to downsample the feature map to reduce the size of the feature map while retaining important feature information. The shape of the feature map obtained after pooling is (112, 112, 32).

[0077] In convolutional layer 2, 64 3x3 convolution kernels are used to convolve the output of pooling layer 1 to further extract mid-level features of the image. The shape of the feature map obtained after convolution is (112, 112, 64).

[0078] In the ReLU activation layer 2, the ReLU activation function is applied to the output of the convolution layer 2, and batch normalization is performed on the output of the ReLU activation layer 2.

[0079] In pooling layer 2, a 2x2 maximum pooling operation is used to downsample the feature map, and the resulting feature map shape is (56, 56, 64).

[0080] In convolutional layer 3, 128 3x3 convolution kernels are used to perform convolution operation on the output of pooling layer 2 to extract high-level features of the image. The shape of the feature map obtained after convolution is (56, 56, 128).

[0081] In the ReLU activation layer 3, the ReLU activation function is applied to the output of the convolution layer 3, and the output of the ReLU activation layer 3 is batch normalized.

[0082] In the pooling layer 3, a 2x2 maximum pooling operation is used to downsample the feature map. The resulting feature map shape is (28, 28, 128). The three-dimensional feature map output by the pooling layer 3 is then flattened into a one-dimensional vector through the flattening layer for input into the fully connected layer.

[0083] Finally, the fully connected layer maps the flattened vector to a 128-dimensional feature space for further feature extraction and classification. The output layer then uses the Softmax activation function to output the probability distribution of the two categories (the subject character exists / the subject character does not exist).

[0084] Step S202: extracting features from the audio stream data using the speech recognition sub-model to obtain sound features, and obtaining scene audio information based on the sound features.

[0085] It is understood that the speech recognition sub-model can be constructed based on a recurrent neural network (RNN), a long short-term memory network (LSTM), or a Transformer model. The speech recognition sub-model can extract sound features from the audio stream data, such as pitch, rhythm, volume, and speech tiling characteristics. Using these extracted sound features, the speech recognition sub-model can identify audio information in the audio, including the presence of speech, speaker characteristics, and background sound type.

[0086] It should also be noted that in order to determine whether the current scene is a scene that needs to highlight the human voice (character scene), it is possible to judge whether there is character voice in the current scene based on the sound characteristics, obtain the character voice judgment result, and then determine the scene audio information based on the character voice judgment result. The scene audio information may be the presence of character voice or the absence of character voice.

[0087] Exemplarily, the speech recognition sub-model constructed based on LSTM may include: 1 input layer, 1 LSTM layer, 1 Dropout layer, 1 fully connected layer and 1 output layer.

[0088] The process of extracting features from the audio stream data using the speech recognition sub-model may be:

[0089] First, the input layer receives the spectral features of the audio stream data, which has the shape of (time steps, feature dimensions), for example (100 frames, 40 dimensions per frame).

[0090] Next, the temporal dependencies in the sequence data are captured through the LSTM layer. This LSTM layer contains 128 LSTM units and can effectively process and predict long-term dependencies in time series data.

[0091] The Dropout operation is then applied to the output of the LSTM layer with a dropout ratio of 0.5 to prevent overfitting.

[0092] Finally, the fully connected layer maps the output of the LSTM layer to a 64-dimensional feature space, and then the output layer uses the Softmax activation function to output the probability distribution of two categories (presence of human voice / absence of human voice).

[0093] Step S203: receiving the scene character information and the scene audio information through the classification output module, and determining the scene category of the current scene according to the scene character information and the scene audio information, wherein the scene categories include: character scene, scenery scene and fusion scene.

[0094] It should be understood that the classification output module can use a classifier, such as a multi-layer perceptron (MLP), a support vector machine (SVM), a decision tree or a neural network, to perform comprehensive feature extraction based on scene character information and scene audio information, and then determine the category of the current scene based on the comprehensive features.

[0095] It should be noted that the scene categories include: human scenes (mainly human activities), scenery scenes (mainly natural or man-made environments) and fusion scenes (both human and scenery information are prominent).

[0096] Exemplarily, a classification output module constructed based on MLP may include: 1 input layer, 1 feature concatenation layer, 2 fully connected layers, 1 Dropout layer and 1 output layer.

[0097] The process of determining the scene category of the current scene according to the scene character information and the scene audio information by the classification output module may be:

[0098] First, the input layer receives 128-dimensional visual features from the person recognition sub-model and 64-dimensional audio features from the speech recognition sub-model.

[0099] Next, the feature concatenation layer concatenates the visual features and audio features into a 192-dimensional fused feature vector.

[0100] The fully connected layer 1 maps the fused features to a 256-dimensional feature space and uses the ReLU activation function to increase the nonlinear expression capability of the model.

[0101] The Dropout layer applies the Dropout operation on the output of the fully connected layer 1 with a dropout ratio of 0.5 to prevent overfitting.

[0102] The fully connected layer 2 maps the 256-dimensional feature vector to a 128-dimensional feature space to further extract fusion features.

[0103] Finally, the output layer uses the Softmax activation function to output the probability distribution of three categories (person scene, scenery scene, and fusion scene).

[0104] This embodiment uses the character recognition sub-model to perform feature extraction on the image frame data to obtain visual features, and obtains scene character information based on the visual features; and uses the speech recognition sub-model to perform feature extraction on the audio stream data to obtain sound features, and obtains scene audio information based on the sound features; thereby, the scene character information and the scene audio information can be received by the classification output module, and the scene category of the current scene can be determined based on the scene character information and the scene audio information, and the scene categories include: character scene, scenery scene, and fusion scene. This embodiment adopts a multimodal fusion approach to integrate visual and auditory information, further improving the accuracy and comprehensiveness of scene recognition.

[0105] Based on the first and second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the first and second embodiments can be referred to above and will not be described in detail. Figure 3 , Figure 3 This is a flow chart of the third embodiment of the audio noise reduction method based on the AI model of this application.

[0106] In this embodiment, in order to specifically explain how to determine the scene category of the current scene based on the combination of different scene character information and the scene audio information, step S203 includes: steps S2031 to S2034:

[0107] Step S2031: When the scene character information indicates that a main character exists and the scene audio information indicates that a character's voice exists, the facial features of the main character are obtained according to the visual features.

[0108] It should be noted that when there is a main character in the current scene and there is character voice in the current scene, it can be regarded that the current scene meets the prerequisite of the character scene.

[0109] It should be understood that after the prerequisites are met, in order to perform more detailed scene division, the facial features of the main characters in the scene can be further extracted based on the visual features obtained above, so as to subsequently determine whether the main character is the speaker in the scene through facial features.

[0110] Step S2032: Determine whether the facial features are consistent with the voice features.

[0111] It should be understood that consistency testing can be performed by checking whether facial features (such as lip shape) and sound features (specific content of the voice) match, that is, judging whether the main character has speaking behavior, whether there is corresponding sound produced when the main character speaks, and whether the main character's lip shape is consistent with the sound produced.

[0112] In a specific implementation, facial features (such as key points of the mouth) and speech features (such as speech spectrum) of the main character can be extracted from the visual features, and whether the main character is speaking can be judged based on the facial features; then dynamic time warping (DTW) or cosine similarity is used to calculate the consistency between the two; finally, the calculated result is compared with a preset threshold. If it is lower than the preset threshold, it is determined that the main character's lip shape is inconsistent with the sound produced.

[0113] Step S2033: If there is consistency, the current scene is determined to be a character scene.

[0114] Step S2034: If there is no consistency, the current scene is determined to be a fusion scene.

[0115] It should be understood that if the facial features are consistent with the voice features, it indicates that the main character is the speaker of the character's voice, that is, the main character is speaking, and the current scene can be determined to be a character scene.

[0116] If the facial features are inconsistent with the voice features, there may be other non-human sounds (such as background music, animal calls, etc.) in the current scene. In this case, the scene is determined to be a fusion scene.

[0117] For example, in a speech scene, the person recognition sub-model detects a speaker (the main person) at a podium, while the speech recognition sub-model captures the clear speech sound (the person's voice). By comparing, it is found that the speaker's mouth shape is synchronized with the speech sound, indicating that the facial features and voice features are consistent, so the scene is determined to be a person scene.

[0118] In the park scene, although the person recognition sub-model detected multiple people (the main characters), the voice recognition captured the sound of birdsong rather than human conversation. At this time, the facial features and voice features are inconsistent, and the system determines that the scene is a fusion scene, which contains the dual elements of people and natural environment.

[0119] Furthermore, considering that the current scene does not meet the prerequisite of the character scene, step S203 further includes steps S2031' to S2032':

[0120] Step S2031 ′: when the scene character information indicates that there is no main character and the scene audio information indicates that there is character voice, if the volume of the character voice is greater than a preset volume value, determining that the current scene is a fusion scene.

[0121] It should be noted that if the scene task information indicates that there are no main characters, but the scene audio information still contains character voices and the volume of the character voices exceeds a set threshold, the current scene can also be determined to be a fusion scene. For example, this fusion scene can include commentary elements such as character interaction or narration.

[0122] Step S2032 ′: when the scene audio information shows that there is no human voice, determining that the current scene is a scene of scenery.

[0123] It is understood that if no human voice is present in the scene audio information, the noise reduction device may directly determine the current scene as a landscape scene, regardless of whether a human (main character) is visually present. For example, a landscape scene may include natural scenery, still life displays, etc., with ambient sound as the dominant factor and lacking interference from human sounds, thereby emphasizing the natural scenery.

[0124] Through the above-mentioned sophisticated scene classification logic, the noise reduction device can more accurately identify and classify scenes in the video, providing more detailed guidance for the subsequent selection of noise reduction strategies based on the scene recognition results, ensuring that each scene can obtain the most suitable noise reduction processing effect.

[0125] Further, in order to specifically illustrate the execution logic of the noise reduction strategy, step S30 specifically includes: steps S301 to S303:

[0126] Step S301: When the current scene is a character scene, a voice enhancement strategy is executed to perform noise reduction processing on the audio stream data to obtain an audio output signal. The voice enhancement strategy is a strategy for enhancing the voice of the main character while reducing the ambient sound.

[0127] It should be understood that the human voice enhancement strategy can be: separating the human voice and the ambient sound through a pre-trained speech separation model (such as Conv-TasNet), then using spectral subtraction to reduce the spectral amplitude of the ambient sound, and finally using adaptive filtering to further suppress the ambient sound.

[0128] In practice, when the current scene is primarily focused on conversation, the noise reduction device can apply an algorithm to enhance the human voice while suppressing background noise. Spectral analysis identifies and amplifies the speaker's voice, while reducing the volume of non-speech frequencies to ensure the listener can clearly hear the speaker. For example, in a lecture scenario, the algorithm increases the speaker's microphone volume while reducing the impact of ambient noise such as air conditioners and fans, allowing the audience to focus on the content.

[0129] Step S302: When the current scene is a landscape scene, an ambient sound enhancement strategy is executed to perform noise reduction processing on the audio stream data to obtain an audio output signal, wherein the ambient sound enhancement strategy is a strategy for enhancing only the ambient sound.

[0130] It should be understood that the ambient sound enhancement strategy can be: using a pre-trained background noise suppression model (such as DC-UNet) to identify and suppress background noise in the audio stream data, preserving the natural characteristics of the ambient sound; then performing dynamic range compression on the ambient sound to enhance low-volume ambient sound and make it more audible; and finally using spectrum enhancement technology to increase the spectrum amplitude of the ambient sound. In addition, echo cancellation technology can also be applied to remove the echo component in the ambient sound and improve the purity of the audio.

[0131] In practice, for scenes without dominant figures, such as natural scenery or city nightscapes, noise reduction equipment can focus its audio processing on enhancing ambient sounds: including natural sounds, wind, and flowing water. This creates a richer, more realistic soundscape, giving viewers an immersive experience. For example, in a forest video, noise reduction equipment can enhance natural background sounds such as rustling leaves and birdsong.

[0132] Step S303: When the current scene is a fusion scene, a fusion enhancement strategy is executed to perform noise reduction processing on the audio stream data to obtain an audio output signal. The fusion enhancement strategy is a strategy for simultaneously enhancing the character voice and the ambient sound.

[0133] It should be understood that the fusion enhancement strategy can be: first, multi-channel processing is performed on the audio stream data to enhance the spectral amplitude of the character voice and the ambient sound respectively, and then the filter parameters are dynamically adjusted according to the spectral characteristics of the character voice and the ambient sound to achieve a balance between the two; then, through the dynamic balance algorithm, the gains of the two are automatically adjusted according to the relative strength of the character voice and the ambient sound in the scene to ensure that both are clearly audible. In addition, in a stereo or surround sound environment, spatial audio enhancement technology can also be applied to create a more three-dimensional and immersive sound experience.

[0134] In practice, in scenarios where both human voices and ambient sounds coexist, noise reduction equipment can enhance both voices and ambient sounds simultaneously, ensuring both are clearly audible while maintaining an appropriate balance to avoid one being too dominant. Fusion enhancement strategies can incorporate more advanced audio processing technologies, such as adaptive filtering and dynamic range control, to achieve the best possible listening experience. For example, in an outdoor concert, noise reduction equipment can harmoniously blend the singer's voice with the sounds of instrumental music while retaining an appropriate amount of ambient sound, such as the cheering of the crowd and distant traffic, creating a rich, layered sound effect that makes the overall experience more three-dimensional and realistic.

[0135] This embodiment uses sophisticated scene classification logic to more accurately identify and categorize scenes in videos, specifically into scenes with people, scenes with objects, and scenes with fusion. It then applies corresponding noise reduction strategies based on these different scene categories, ensuring that each scene receives the most appropriate noise reduction treatment. This improves the audiovisual quality of live video and provides viewers with a more personalized and immersive viewing experience.

[0136] It should be noted that the above examples are only used to understand this application and do not constitute a limitation on the audio noise reduction method based on the AI model of this application. More simple transformations based on this technical concept are all within the scope of protection of this application.

[0137] In addition, this application also provides an audio noise reduction device, referring to Figure 4 , Figure 4 This is a schematic diagram of the module structure of the first embodiment of the audio noise reduction system based on the AI model of this application; Figure 4 As shown, the system includes:

[0138] The video segmentation module 401 is used to receive a live video stream and segment the live video stream into image frame data and audio stream data;

[0139] A scene recognition module 402 is configured to input the image frame data and the audio stream data into a preset scene recognition model to obtain a recognition result of the current scene, wherein the preset scene recognition model includes a person recognition sub-model, a speech recognition sub-model, and a classification output module, wherein the person recognition sub-model is configured to determine scene person information based on the image frame data, the speech recognition sub-model is configured to determine scene audio information based on the audio stream data, and the classification output module is configured to determine the scene category of the current scene based on the scene person information and the scene audio information;

[0140] The strategy matching module 403 is used to determine and execute the current noise reduction strategy according to the recognition result to obtain an audio output signal. The current noise reduction strategy is: human voice enhancement strategy, ambient sound enhancement strategy or fusion enhancement strategy.

[0141] Furthermore, the scene recognition module 402 is also used to perform feature extraction on the image frame data through the character recognition sub-model to obtain visual features, and obtain scene character information based on the visual features; perform feature extraction on the audio stream data through the speech recognition sub-model to obtain sound features, and obtain scene audio information based on the sound features; receive the scene character information and the scene audio information through the classification output module, and determine the scene category of the current scene based on the scene character information and the scene audio information, and the scene categories include: character scene, scenery scene and fusion scene.

[0142] Furthermore, the scene recognition module 402 is also used to determine whether there is a main character in the current scene based on the visual features, and obtain a main character judgment result; obtain scene character information based on the main character judgment result, and the scene character information is: whether there is a main character or not.

[0143] Furthermore, the scene recognition module 402 is also used to determine whether there is human voice in the current scene based on the sound characteristics, and obtain a human voice judgment result; obtain scene audio information based on the human voice judgment result, and the scene audio information is: there is human voice or there is no human voice.

[0144] Furthermore, the scene recognition module 402 is also used to obtain the facial features of the main character based on the visual features when the scene character information indicates that there is a main character and the scene audio information indicates that there is a character voice; determine whether there is consistency between the facial features and the sound features; if there is consistency, determine that the current scene is a character scene; if there is no consistency, determine that the current scene is a fusion scene.

[0145] Furthermore, the scene recognition module 402 is also used to determine that the current scene is a fusion scene when the scene character information shows that there is no main character and the scene audio information shows that there is character voice, if the volume of the character voice is greater than a preset volume value; and when the scene audio information shows that there is no character voice, determine that the current scene is a scenery scene.

[0146] Furthermore, the strategy matching module 403 is also used to, when the current scene is a character scene, execute the human voice enhancement strategy to perform noise reduction processing on the audio stream data to obtain an audio output signal, and the human voice enhancement strategy is a strategy for enhancing the character voice of the main character while reducing the ambient sound; when the current scene is a landscape scene, execute the ambient sound enhancement strategy to perform noise reduction processing on the audio stream data to obtain an audio output signal, and the ambient sound enhancement strategy is a strategy for only enhancing the ambient sound; when the current scene is a fusion scene, execute the fusion enhancement strategy to perform noise reduction processing on the audio stream data to obtain an audio output signal, and the fusion enhancement strategy is a strategy for simultaneously enhancing the character voice and the ambient sound.

[0147] This embodiment uses a multimodal fusion approach to identify scenes in videos by combining image and audio information. This improves the robustness of scene recognition in complex or changing environments. Based on the scene recognition results, different noise reduction strategies are dynamically implemented, such as voice enhancement, scene sound enhancement, or fusion enhancement. This adaptively adjusts the noise reduction strategy based on the current scene, improving audio output quality in different scenarios.

[0148] Other embodiments or specific implementations of the AI model-based audio noise reduction system described in this application can refer to the above-mentioned method embodiments and will not be repeated here.

[0149] The present application also provides an audio noise reduction device based on an AI model, and the audio noise reduction device based on the AI model includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the audio noise reduction method based on the AI model in the above-mentioned embodiment one.

[0150] Reference below Figure 5 , Figure 5 The following is a schematic diagram of the structure of the audio noise reduction device based on the AI model of the present application. The audio noise reduction device based on the AI model in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The AI model-based audio noise reduction device shown is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

[0151] like Figure 5As shown, the audio noise reduction device based on the AI model may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the audio noise reduction device based on the AI model are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the audio noise reduction device based on the AI model to communicate wirelessly or wired with other devices to exchange data. Although the figure shows an audio noise reduction device based on the AI model with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0152] The present application also provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, and the computer-readable program instructions are used to execute the AI model-based audio noise reduction method in the above-mentioned embodiment.

[0153] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0154] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An audio noise reduction method based on an AI model, characterized in that: The method comprises: Receive a live video stream, and split the live video stream into image frame data and audio stream data; Inputting the image frame data and the audio stream data into a preset scene recognition model to obtain a recognition result of the current scene, wherein the preset scene recognition model includes a person recognition sub-model, a speech recognition sub-model, and a classification output module, wherein the person recognition sub-model is used to determine scene person information based on the image frame data, the speech recognition sub-model is used to determine scene audio information based on the audio stream data, and the classification output module is used to determine the scene category of the current scene based on the scene person information and the scene audio information; A current noise reduction strategy is determined and executed according to the recognition result to obtain an audio output signal, wherein the current noise reduction strategy is: a human voice enhancement strategy, an ambient sound enhancement strategy, or a fusion enhancement strategy.

2. The method according to claim 1, wherein The step of inputting the image frame data and the audio stream data into a preset scene recognition model to obtain a recognition result of the current scene includes: Extracting features from the image frame data using the character recognition sub-model to obtain visual features, and obtaining scene character information based on the visual features; Extracting features from the audio stream data using the speech recognition sub-model to obtain sound features, and obtaining scene audio information based on the sound features; The classification output module receives the scene character information and the scene audio information, and determines the scene category of the current scene according to the scene character information and the scene audio information. The scene categories include: character scene, scenery scene and fusion scene.

3. The method according to claim 2, wherein The step of obtaining scene character information according to the visual features includes: Determine whether there is a subject person in the current scene based on the visual features, and obtain a subject person determination result; The scene character information is obtained according to the main character judgment result, and the scene character information is: the main character exists or does not exist.

4. The method according to claim 3, wherein The step of obtaining scene audio information according to the sound characteristics includes: Determine whether there is a human voice in the current scene based on the sound characteristics, and obtain a human voice determination result; The scene audio information is obtained according to the character voice judgment result, and the scene audio information is: the presence of the character voice or the absence of the character voice.

5. The method according to claim 4, wherein The step of determining the scene category of the current scene based on the scene character information and the scene audio information includes: When the scene character information indicates that a subject character exists and the scene audio information indicates that a character voice exists, obtaining facial features of the subject character according to the visual features; Determining whether the facial features are consistent with the voice features; If there is consistency, determining that the current scene is a character scene; If there is no consistency, the current scene is determined to be a fusion scene.

6. The method according to claim 4, wherein The step of determining the scene category of the current scene according to the scene character information and the scene audio information further includes: When the scene character information indicates that there is no main character and the scene audio information indicates that there is a character voice, if the volume of the character voice is greater than a preset volume value, determining that the current scene is a fusion scene; When the scene audio information shows that there is no human voice, it is determined that the current scene is a landscape scene.

7. The method according to claim 2, wherein The step of determining and executing the current noise reduction strategy according to the recognition result to obtain the audio output signal includes: When the current scene is a character scene, executing a human voice enhancement strategy to perform noise reduction processing on the audio stream data to obtain an audio output signal, wherein the human voice enhancement strategy is a strategy for enhancing the voice of the main character while reducing the ambient sound; When the current scene is a landscape scene, executing an ambient sound enhancement strategy to perform noise reduction processing on the audio stream data to obtain an audio output signal, wherein the ambient sound enhancement strategy is a strategy for enhancing only the ambient sound; When the current scene is a fusion scene, a fusion enhancement strategy is executed to perform noise reduction processing on the audio stream data to obtain an audio output signal. The fusion enhancement strategy is a strategy for simultaneously enhancing character voices and ambient sounds.

8. An audio noise reduction system based on an AI model, characterized in that: The system comprises: A video segmentation module is used to receive a live video stream and split the live video stream into image frame data and audio stream data; a scene recognition module, configured to input the image frame data and the audio stream data into a preset scene recognition model to obtain a recognition result of the current scene, wherein the preset scene recognition model includes a person recognition sub-model, a speech recognition sub-model, and a classification output module, wherein the person recognition sub-model is configured to determine scene person information based on the image frame data, the speech recognition sub-model is configured to determine scene audio information based on the audio stream data, and the classification output module is configured to determine the scene category of the current scene based on the scene person information and the scene audio information; The strategy matching module is used to determine and execute the current noise reduction strategy according to the recognition result to obtain an audio output signal, where the current noise reduction strategy is: human voice enhancement strategy, ambient sound enhancement strategy or fusion enhancement strategy.

9. An audio noise reduction device based on an AI model, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the audio noise reduction method based on the AI model as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the audio noise reduction method based on the AI model are implemented as described in any one of claims 1 to 7.

Images