Audio and video synchronous noise reduction method and system based on AI visual perception

Abstract

The application relates to the technical field of video data processing, and relates to an audio-video synchronous noise reduction method and system based on AI visual perception, which comprises the following steps: pre-processing audio data to obtain pre-processed audio data; performing time-frequency analysis on the pre-processed audio data to obtain a speech feature set and a background sound feature set; performing noise reduction on video data to obtain primary noise reduction video data, performing visual perception on the primary noise reduction video data to obtain a video feature set; performing time axis correction operation on the speech feature set based on a mouth shape feature according to the video feature set to obtain an updated time axis; performing active noise reduction operation on the progress correction audio data according to the updated time axis and a pre-constructed background sound adaptation degree sequence to obtain noise correction audio data; and performing merging operation on the noise correction audio data and the primary noise reduction video data to obtain synchronous noise reduction audio-video. The application can improve the clarity of images and sounds in a video.

Description

Technical Field

[0001] This invention relates to the field of video data processing technology, and in particular to an audio-visual synchronization noise reduction method and system based on AI visual perception. Background Technology

[0002] With advancements in recording and video equipment technology, users have increasingly higher demands for audio and video quality. Modern devices excel at capturing details, but they are also more prone to capturing ambient noise, such as environmental noise and wind noise.

[0003] In addition, in terms of data transmission, audio and video may suffer from transmission errors, data compression, and data corruption, resulting in problems such as image distortion, excessive noise, and audio-visual asynchrony, which degrade audio and video quality and reduce the user's viewing experience. Summary of the Invention

[0004] This invention provides an audio-visual synchronization noise reduction method based on AI visual perception, the main purpose of which is to improve the clarity of images and sounds in videos.

[0005] To achieve the above objectives, this invention provides an audio-visual synchronization noise reduction method based on AI visual perception, comprising:

[0006] Acquire audio and video data, perform audio-visual separation on the audio and video data to obtain audio data and video data;

[0007] The audio data is filtered using a pre-built low-pass filter to obtain filtered audio data, and then downsampled according to a preset sampling rate to obtain preprocessed audio data.

[0008] The preprocessed audio data is subjected to time-frequency analysis using a pre-constructed wavelet transform algorithm to obtain an audio signal feature set. Then, an audio separation operation is performed on the audio signal feature set to classify speech audio and background audio, resulting in a speech feature set and a background audio feature set.

[0009] Using a pre-trained audio-visual perception denoising model, the video data is denoised using a pre-built optimized minimum mean square error algorithm to obtain primary denoised video data. Then, the primary denoised video data is subjected to object and scene-based visual perception operations to obtain a video feature set.

[0010] Based on the video feature set, a timeline correction operation based on lip shape features is performed on the speech feature set to obtain an updated timeline;

[0011] According to the update timeline, a progress update operation is performed on the background sound feature set to obtain an updated background sound feature set. Then, according to the video feature set, a scene adaptability recognition operation is performed on the updated background sound feature set to output the adaptability score of each background sound in the updated background sound feature set in each time period, thus obtaining a background sound adaptability sequence.

[0012] The audio data is progress-corrected according to the update timeline to obtain progress-corrected audio data. Active noise reduction is performed on the progress-corrected audio data according to the background sound adaptation sequence to obtain noise-corrected audio data. The noise-corrected audio data and the primary noise-reduced video data are then time-synchronized audio-visual merging operations to obtain synchronized noise-reduced video.

[0013] Optionally, the step of performing time-frequency analysis on the preprocessed audio data using a pre-constructed wavelet transform algorithm to obtain an audio signal feature set includes:

[0014] Obtain the Haar wavelet, and perform translation and scaling operations on the Haar wavelet to obtain the wavelet function;

[0015] The preprocessed audio data and the wavelet function are convolved to obtain wavelet coefficients, wherein the wavelet coefficients are expressed as:

[0016]

[0017] In the formula, W(a, b) represents wavelet coefficients, a represents scale frequency, b represents position time, t represents time, and x(t) represents the preprocessed audio data. Represents the wavelet function;

[0018] The wavelet coefficients are reconstructed using a pre-constructed reconstruction formula to obtain an updated wavelet graph, wherein the updated wavelet graph is represented as follows:

[0019]

[0020] In the formula, x'(t) represents the updated wavelet graph, j represents the new scale frequency, k represents the new location time, and W j (k) represents the wavelet coefficients at scale frequency j and location time k. Represents the wavelet function at scale frequency j and location time k;

[0021] The updated wavelet graph is subjected to time- and frequency-based signal feature extraction operations to obtain an audio signal feature set.

[0022] Optionally, the audio separation operation of classifying the audio signal feature set into speech audio and background audio to obtain the speech feature set and the background audio feature set includes:

[0023] Perform fully connected audio object recognition on the audio signal feature set to obtain an audio object set;

[0024] The audio object set is classified to obtain a speech feature set and a background sound feature set. The speech feature set includes the speech of characters in the video, the speech of characters outside the video, and the speech of the video narration. The background sound feature set is the sound type in the audio object set that is not in the speech feature set.

[0025] Optionally, before utilizing the pre-trained audiovisual perception noise reduction model, the method further includes:

[0026] Obtain an audiovisual perception denoising model that includes a video perception network, a sound perception network, an active denoising network, and an audio-visual output network; obtain a training sample set, wherein the video perception network and the sound perception network are networks to be trained, the active denoising network and the audio-visual output network are networks with fixed parameters, and the training sample set includes sample audio and video, as well as video object labels, audio object labels and manual correction labels corresponding to the sample audio and video.

[0027] A target sample is extracted sequentially from the training sample set, and the audio-visual perception noise reduction model is used to perform forward network calculation on the target sample. The pre-built intermediate layer output interface is used to obtain the predicted video object recognition result, the predicted audio object recognition result, and the predicted time axis deviation of the forward network calculation process.

[0028] Using a pre-built cross-entropy loss algorithm, a first loss value is calculated between the predicted video object recognition result and the video object label; a second loss value is calculated between the predicted audio object recognition result and the audio object label; and a third loss value is calculated between the predicted time axis deviation and the manually corrected label.

[0029] Based on the pre-built gradient descent algorithm, the first loss value, the second loss value and the third loss value are minimized to obtain a set of sample parameters. The set of sample parameters is then updated in reverse to obtain an updated audio-visual perception noise reduction model.

[0030] Based on the pre-built model ensemble algorithm, the model parameters of the updated audio-visual perception denoising model corresponding to multiple target samples are obtained, resulting in a set of model parameters. The mean of the set of model parameters is then calculated to obtain the trained audio-visual perception denoising model.

[0031] Optionally, the step of performing noise reduction on the video data based on a pre-constructed optimized minimum mean square error algorithm to obtain primary noise-reduced video data includes:

[0032] By obtaining the filter and the minimum mean square error algorithm, the objective function for the minimum mean square error is obtained, wherein the objective function is expressed as:

[0033] J(w)=E{e 2 [n]}=E{(d[n]-y[n]) 2}

[0034] In the formula, J(w) represents the objective function, w represents the coefficients of the filter, E{·} represents the expectation operator, and e 2 [n] represents the square of the error signal, d[n] represents the desired signal of the filter, and y[n] represents the output signal of the filter;

[0035] Adaptive regularization is introduced into the minimum mean square error algorithm to obtain the optimization objective function, which is expressed as:

[0036] J′(w)=E{e 2 [n]}+γR(w)

[0037]

[0038] In the formula, J'(w) represents the objective function, γ represents the regularization parameter, and R(w) represents the L2 norm regularization term. Represents the square of the Euclidean norm;

[0039] Obtain the coefficients of the filter, and adjust the coefficients according to the optimization objective function to obtain updated filter coefficients, wherein the updated filter coefficients are expressed as:

[0040] w[n+1]=w[n]+μe[n]x[n]-2γw[n]

[0041] In the formula, w[n]+1] represents the updated filter coefficients, w[n] represents the initial filter coefficients, μ represents the weight coefficients, and 2γw[n] represents the gradient descent term of the L2 norm regularization term;

[0042] The video data is filtered according to the updated filtering coefficients to obtain primary noise-reduced video data.

[0043] Optionally, the step of performing object- and scene-based visual perception operations on the initial denoised video data to obtain a video feature set includes:

[0044] Perform a convolution operation on the primary noise-reduced video data to obtain a set of video convolution matrices;

[0045] The set of video convolutional matrices is subjected to dimensionality reduction to obtain a set of video feature sequences;

[0046] Perform a fully connected recognition operation based on objects and scenes on the video feature sequence set to obtain the video feature set.

[0047] Optionally, the step of performing a timeline correction operation based on lip-sync features on the speech feature set according to the video feature set to obtain an updated timeline includes:

[0048] Perform lip-shape feature recognition on the speech feature set to obtain the speech lip-shape recognition result and the time axis corresponding to the speech lip-shape recognition result;

[0049] Based on the video feature set and the speech lip-sync recognition results, the timeline is subjected to recognition operations based on forward / backward and scaling to obtain the timeline deviation;

[0050] The time axis is updated based on the time axis deviation to obtain an updated time axis.

[0051] Optionally, the step of performing active noise reduction on the progress-corrected audio data based on the background sound adaptation sequence to obtain noise-corrected audio data includes:

[0052] Background sound types with fit scores less than a preset noise interference threshold in the background sound fit sequence are extracted to obtain noise background sound types;

[0053] The progress-corrected audio data is subjected to object type identification to obtain an audio object result set;

[0054] The background sounds that match the noise background sound type in the audio object result set are reversed to obtain the reverse audio set;

[0055] Using the inverse audio set, a signal superposition operation is performed on the progress-corrected audio data to obtain noise-corrected audio data.

[0056] Optionally, after obtaining the synchronized noise-reduced video, the method further includes:

[0057] The background sound fit sequence is visualized to obtain a noise removal list;

[0058] Utilize a pre-built user feedback service to obtain users' noise reduction needs;

[0059] Based on the noise reduction requirements, the background sound adaptation sequence and noise removal list are modified to obtain a synchronized noise-reduced video that meets the user's needs.

[0060] To achieve the above objectives, the present invention also provides an audio-visual synchronization noise reduction system based on AI visual perception, comprising:

[0061] The raw data acquisition module is used to acquire audio and video, and perform audio-visual separation operation on the audio and video to obtain audio data and video data;

[0062] The audio feature recognition module is used to filter the audio data using a pre-built low-pass filter to obtain filtered audio data, and to downsample the filtered audio data according to a preset sampling rate to obtain pre-processed audio data. It also uses a pre-built wavelet transform algorithm to perform time-frequency analysis on the pre-processed audio data to obtain an audio signal feature set, and performs audio separation operation to classify the audio signal feature set into speech audio and background audio to obtain a speech feature set and a background audio feature set.

[0063] The audio-visual feature recognition module is used to perform noise reduction on the video data based on a pre-built optimized minimum mean square error algorithm using a pre-trained audio-visual perception noise reduction model to obtain primary noise-reduced video data, and to perform visual perception operations based on objects and scenes on the primary noise-reduced video data to obtain a video feature set.

[0064] The synchronous noise reduction module is used to perform a timeline correction operation based on lip-sync features on the speech feature set according to the video feature set to obtain an updated timeline; and to perform a progress update operation on the background sound feature set according to the updated timeline to obtain an updated background sound feature set. It then performs a scene fit recognition operation on the updated background sound feature set according to the video feature set, outputting the fit score of each background sound in the updated background sound feature set within each time period to obtain a background sound fit sequence. Furthermore, it performs progress correction on the audio data according to the updated timeline to obtain progress-corrected audio data; performs active noise reduction on the progress-corrected audio data according to the background sound fit sequence to obtain noise-corrected audio data; and performs a time-synchronized audio-visual merging operation on the noise-corrected audio data and the primary noise-reduced video data to obtain a synchronous noise-reduced video.

[0065] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0066] Memory, storing at least one instruction; and

[0067] The processor executes the instructions stored in the memory to implement the AI-based visual perception-based audio and video synchronization noise reduction method described above.

[0068] To address the aforementioned issues, the present invention also provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the aforementioned AI-based visual perception-based audio-visual synchronization noise reduction method.

[0069] To address the problems described in the background section, this invention first performs audio-visual separation to obtain independently processable audio and video data. Then, it preprocesses the audio data and performs time-frequency analysis to obtain a speech feature set and a background noise feature set. The speech feature set can be used to match lip movements in the video data to locate the timeline, while the background noise feature set can analyze noise components for targeted noise reduction. Furthermore, this invention utilizes a pre-trained audio-visual perception noise reduction model to first analyze video features to obtain a video feature set, and then analyze the matching degree between video and audio features. This not only allows for timeline location but also the location of unwanted background noise in the audio, enabling active noise reduction. Finally, the denoised audio and video are merged to obtain a synchronously noise-reduced video. Therefore, this invention can improve the clarity of images and sound in videos. Attached Figure Description

[0070] Figure 1 This is a flowchart illustrating an AI-based visual perception-based audio-video synchronization noise reduction method according to an embodiment of the present invention.

[0071] Figure 2 This is a functional block diagram of an audio-visual synchronization noise reduction system based on AI visual perception provided in an embodiment of the present invention;

[0072] Figure 3 This is a schematic diagram of the structure of an electronic device that implements the AI-based visual perception-based audio and video synchronization noise reduction method according to an embodiment of the present invention.

[0073] Figure label explanation:

[0074] 10. Electronic device; 11. Processor; 12. Memory; 13. Bus.

[0075] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0076] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0077] This application provides an AI-based visual perception-based audio-video synchronization noise reduction method. The execution entity of this AI-based visual perception-based audio-video synchronization noise reduction method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application embodiment: a server, a terminal, etc. In other words, the AI-based visual perception-based audio-video synchronization noise reduction method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0078] Reference Figure 1 The diagram shown is a flowchart illustrating an AI-based visual perception-based audio-video synchronization noise reduction method according to an embodiment of the present invention. In this embodiment, the AI-based visual perception-based audio-video synchronization noise reduction method includes:

[0079] S1. Acquire audio and video data, and perform audio-visual separation operation on the audio and video data to obtain audio data and video data.

[0080] The audio and video referred to here refers to the raw data containing audio and video.

[0081] Specifically, in this embodiment of the invention, the audio and video formats are first converted to a format suitable for processing, such as MP4 or AVI format. Then, an audio processing library (such as FFmpeg) is used to extract the audio and video tracks from the audio and video to generate separate audio and video data.

[0082] S2. The audio data is filtered using a pre-built low-pass filter to obtain filtered audio data, and the filtered audio data is downsampled according to a preset sampling rate to obtain pre-processed audio data.

[0083] In this embodiment of the invention, the audio data needs to be preprocessed before analysis and processing.

[0084] A low-pass filter is an electronic filter that allows signals below the cutoff frequency to pass through, but prevents signals above the cutoff frequency from passing through.

[0085] Furthermore, downsampling is a technique in audio preprocessing used to reduce the sampling rate of audio signals in digital signal processing, thereby reducing the amount of data and computational burden.

[0086] Specifically, according to the Nyquist sampling theorem, to prevent aliasing, the sampling rate after downsampling must be greater than twice the highest frequency component of the original signal. Therefore, before downsampling, a low-pass filter is usually required to remove frequency components higher than half of the new sampling rate.

[0087] Therefore, in this embodiment of the invention, the audio data is first filtered using a low-pass filter to obtain filtered audio data. Once the signal has been filtered, the sampling rate can be reduced according to a set ratio. For example, if the sampling rate is reduced from 44.1kHz to 22.05kHz, it actually means that one sample is retained for every two samples, resulting in preprocessed audio data. The sampling rate is related to multiple factors such as the nature of the audio signal, the application scenario, and the requirements of subsequent processing. For example, the standard sampling rate for CD audio quality is 44.1kHz, while for film and television works it is 48kHz, etc. In this embodiment of the invention, the sampling rate can be selected according to the type of audio and video.

[0088] S3. Use a pre-constructed wavelet transform algorithm to perform time-frequency analysis on the preprocessed audio data to obtain an audio signal feature set, and perform audio separation operation to classify the audio signal feature set into speech audio and background audio to obtain a speech feature set and a background audio feature set.

[0089] In detail, in this embodiment of the invention, the step of using a pre-constructed wavelet transform algorithm to perform time-frequency analysis on the preprocessed audio data to obtain an audio signal feature set includes:

[0090] Obtain the Haar wavelet, and perform translation and scaling operations on the Haar wavelet to obtain the wavelet function;

[0091] The preprocessed audio data and the wavelet function are convolved to obtain wavelet coefficients, wherein the wavelet coefficients are expressed as:

[0092]

[0093] In the formula, W(a, b) represents wavelet coefficients, a represents scale frequency, b represents position time, t represents time, and x(t) represents the preprocessed audio data. Represents the wavelet function;

[0094] The wavelet coefficients are reconstructed using a pre-constructed reconstruction formula to obtain an updated wavelet function, wherein the updated wavelet graph is represented as follows:

[0095]

[0096] In the formula, x'(t) represents the updated wavelet graph, j represents the new scale frequency, k represents the new location time, and W j (k) represents the wavelet coefficients at scale frequency j and location time k. Represents the wavelet function at scale frequency j and location time k;

[0097] The updated wavelet graph is subjected to time- and frequency-based signal feature extraction operations to obtain an audio signal feature set.

[0098] The wavelet transform algorithm decomposes the signal into components of different frequencies, enabling simultaneous analysis of signal characteristics in both the time and frequency domains. In this embodiment, time-frequency analysis using wavelet transform offers better time-frequency localization capabilities than traditional Fourier transform. Translation refers to the movement of the wavelet function along the time axis, while scaling refers to changing the scale of the wavelet function to analyze the signal at different resolutions.

[0099] Furthermore, a wavelet function is a waveform with localization characteristics, typically possessing finite time support and tunable frequency characteristics. The Haar wavelet used in this embodiment is the simplest type of wavelet transform, belonging to the discrete wavelet transform category, and its mathematical form is simple and easy to implement. The reconstruction formula refers to... This represents the cumulative frequency and location time at various different scales.

[0100] Specifically, in this embodiment of the invention, wavelet decomposition is first performed using the initialized scale frequency and location time, and then reconstructed based on the new scale frequency and location time to obtain an updated wavelet map. The updated wavelet map not only shows the signal features at different scales, but also shows the frequency changes of the signal within a specific time period. Therefore, time-frequency analysis of the updated wavelet map can extract features from both time and frequency aspects to obtain an audio signal feature set.

[0101] In detail, in this embodiment of the invention, the audio separation operation of classifying the audio signal feature set into speech audio and background audio to obtain the speech feature set and the background audio feature set includes:

[0102] Perform fully connected audio object recognition on the audio signal feature set to obtain an audio object set;

[0103] The audio object set is classified to obtain a speech feature set and a background sound feature set. The speech feature set includes the speech of characters in the video, the speech of characters outside the video, and the speech of the video narration. The background sound feature set is the sound type in the audio object set that is not in the speech feature set.

[0104] Specifically, in this embodiment of the invention, an audio object set can be obtained by recognizing audio objects. The fully connected audio object recognition refers to the fully connected operation in a neural network used to achieve audio object recognition. The fully connected layer is a layered structure where each input node is connected to each output node, used in the last few layers of the neural network, especially in classification tasks, to map features to the final output category.

[0105] The audio object recognition is represented as [character 1 voice, character n voice, ..., fighting sound, wind sound, rain sound, bamboo forest sound, ...]. This invention can then classify the audio object set to obtain a speech feature set and a background sound feature set. For example, the speech feature set could be [character 1 voice, character n voice, narration sound, ...] and the background sound feature set could be [off-stage sound, fighting sound, wind sound, rain sound, bamboo forest sound, ...].

[0106] S4. Using a pre-trained audio-visual perception denoising model, perform denoising operations on the video data based on a pre-built optimized minimum mean square error algorithm to obtain primary denoised video data, and perform object and scene-based visual perception operations on the primary denoised video data to obtain a video feature set.

[0107] In this embodiment of the invention, the audio-visual perception noise reduction model is a neural network-based model used to analyze image features in video data and sound features in audio data, then actively denoise unnecessary sounds in the audio data, and can also denoise video images, thereby outputting synchronously noise-reduced video.

[0108] The audio-visual perception noise reduction model includes a video perception network, a sound perception network, an active noise reduction network, and an audio-visual output network.

[0109] In detail, in this embodiment of the invention, before utilizing the pre-trained audiovisual perception noise reduction model, the method further includes:

[0110] Obtain an audiovisual perception denoising model that includes a video perception network, a sound perception network, an active denoising network, and an audio-visual output network; obtain a training sample set, wherein the video perception network and the sound perception network are networks to be trained, the active denoising network and the audio-visual output network are networks with fixed parameters, and the training sample set includes sample audio and video, as well as video object labels, audio object labels and manual correction labels corresponding to the sample audio and video.

[0111] A target sample is extracted sequentially from the training sample set, and the audio-visual perception noise reduction model is used to perform forward network calculation on the target sample. The pre-built intermediate layer output interface is used to obtain the predicted video object recognition result, the predicted audio object recognition result, and the predicted time axis deviation of the forward network calculation process.

[0112] Using a pre-built cross-entropy loss algorithm, a first loss value is calculated between the predicted video object recognition result and the video object label; a second loss value is calculated between the predicted audio object recognition result and the audio object label; and a third loss value is calculated between the predicted time axis deviation and the manually corrected label.

[0113] Based on the pre-built gradient descent algorithm, the first loss value, the second loss value and the third loss value are minimized to obtain a set of sample parameters. The set of sample parameters is then updated in reverse to obtain an updated audio-visual perception noise reduction model.

[0114] Based on the pre-built model ensemble algorithm, the model parameters of the updated audio-visual perception denoising model corresponding to multiple target samples are obtained, resulting in a set of model parameters. The mean of the set of model parameters is then calculated to obtain the trained audio-visual perception denoising model.

[0115] In this embodiment of the invention, each audio and video sample in the training sample set contains labels for video objects judged by human senses [television, person, cat, rainy day], labels for audio objects [speaking sound, water boiling sound, wind sound, rain sound], and labels for manual correction [audio delayed by 0.5s].

[0116] This invention obtains predicted video object recognition results, predicted audio object recognition results, and predicted time axis deviation by computing a forward network on the target samples. Since the predicted video and audio object recognition results involve multi-object classification, a multi-task cross-entropy loss algorithm is used, while the loss value between the predicted time axis deviation and manually corrected labels is calculated using a single-task cross-entropy loss algorithm. The cross-entropy loss algorithm is used to measure the difference between two probability distributions and is typically used to evaluate the difference between the model's predictions and the true label distribution.

[0117] Furthermore, this invention minimizes each loss value using the gradient descent algorithm. Since the loss values ​​are independent of each other, the model parameters corresponding to each minimized loss value can be uniformly updated in reverse to obtain an updated audio-visual perception noise reduction model. The gradient descent algorithm is an optimization algorithm used to find the optimal values ​​of the model parameters to minimize the loss function.

[0118] Finally, this invention calculates the mean of the network model parameters in each updated audio-visual perception denoising model to obtain a more balanced audio-visual perception denoising model. The mean calculation refers to a weighted average of the parameters obtained during training.

[0119] In this embodiment of the invention, after obtaining the audiovisual perception noise reduction model, video data and audio data can be processed.

[0120] In detail, in this embodiment of the invention, the step of performing noise reduction on the video data based on a pre-constructed optimized minimum mean square error algorithm to obtain primary noise-reduced video data includes:

[0121] By obtaining the filter and the minimum mean square error algorithm, the objective function for the minimum mean square error is obtained, wherein the objective function is expressed as:

[0122] J(w)=E{e 2 [n]}=E{(d[n]-y[n]) 2}

[0123] In the formula, J(w) represents the objective function, w represents the coefficients of the filter, E{·} represents the expectation operator, and e 2 [n] represents the square of the error signal, d[n] represents the desired signal of the filter, and y[n] represents the output signal of the filter;

[0124] Adaptive regularization is introduced into the minimum mean square error algorithm to obtain the optimization objective function, which is expressed as:

[0125] J′(w)=E{e 2 [n]}+γR(w)

[0126]

[0127] In the formula, J′(w) represents the objective function, γ represents the regularization parameter, and R(w) represents the L2 norm regularization term. Represents the square of the Euclidean norm;

[0128] Obtain the coefficients of the filter, and adjust the coefficients according to the optimization objective function to obtain updated filter coefficients, wherein the updated filter coefficients are expressed as:

[0129] w[n+1]=w[n]+μe[n]x[n]-2γw[n]

[0130] In the formula, w[n+1] represents the updated filter coefficients, w[n] represents the initial filter coefficients, μ represents the weight coefficients, and 2γw[n] represents the gradient descent term of the L2 norm regularization term;

[0131] The video data is filtered according to the updated filtering coefficients to obtain primary noise-reduced video data.

[0132] It should be noted that the Least Mean Square (LMS) algorithm, as a classic adaptive filter algorithm, is widely used in the denoising of digital audio signals. However, the traditional LMS algorithm suffers from slow convergence speed and poor stability when dealing with non-stationary noise, making it difficult to meet the needs of practical applications. Therefore, in this embodiment of the invention, a regularization algorithm is introduced based on the LMS algorithm to obtain the optimized LMS algorithm.

[0133] Specifically, in this embodiment of the invention, the filter coefficients of the minimum mean square error algorithm are first updated according to the Euclidean norm, and then data filtering is performed to obtain primary noise-reduced video data.

[0134] The introduction of a regularization term in this embodiment effectively suppresses the influence of noise on filter coefficients, improving the stability and robustness of the algorithm. Furthermore, the introduction of the regularization parameter γ enables the algorithm to adaptively adjust the filter update rules, dynamically adjusting the step size and regularization degree based on the characteristics of the current signal and noise, thereby accelerating convergence and improving noise resistance.

[0135] Furthermore, in this embodiment of the invention, the step of performing object- and scene-based visual perception operations on the primary denoised video data to obtain a video feature set includes:

[0136] Perform a convolution operation on the primary noise-reduced video data to obtain a set of video convolution matrices;

[0137] The set of video convolutional matrices is subjected to dimensionality reduction to obtain a set of video feature sequences;

[0138] Perform a fully connected recognition operation based on objects and scenes on the video feature sequence set to obtain the video feature set.

[0139] Specifically, in this embodiment of the invention, a pre-constructed convolutional kernel layer is used to perform noise reduction on the initial noise-reducing video data. Then, a pooling layer and a flattening layer are used to perform dimensionality reduction on the video convolutional matrix set until it is one dimension, resulting in a video feature sequence set. Finally, this embodiment of the invention performs fully connected recognition on the video feature sequence set to obtain an object set, and identifies the scene in the video based on the combination of each object, thereby comprehensively outputting the objects and the scene to obtain the video feature set.

[0140] S5. Based on the video feature set, perform a timeline correction operation on the speech feature set based on lip shape features to obtain an updated timeline.

[0141] In detail, in this embodiment of the invention, the step of performing a timeline correction operation based on lip-shape features on the speech feature set according to the video feature set to obtain an updated timeline includes:

[0142] Perform lip-shape feature recognition on the speech feature set to obtain the speech lip-shape recognition result and the time axis corresponding to the speech lip-shape recognition result;

[0143] Based on the video feature set and the speech lip-sync recognition results, the timeline is subjected to recognition operations based on forward / backward and scaling to obtain the timeline deviation;

[0144] The time axis is updated based on the time axis deviation to obtain an updated time axis.

[0145] Specifically, in this embodiment of the invention, lip movements are first predicted based on a set of speech features to obtain a lip movement recognition result and a corresponding timeline. Then, audio-visual matching is performed using a set of video features to determine whether the timeline needs to be advanced, regressed, or scaled. For example, the timeline deviation could be [audio forward 0.5s, audio contracted 1%]. Then, the timeline is weighted or summed based on the timeline deviation to obtain an updated timeline.

[0146] S6. According to the update timeline, perform a progress update operation on the background sound feature set to obtain an updated background sound feature set, and according to the video feature set, perform a scene adaptation recognition operation on the updated background sound feature set to output the adaptation score of each background sound in the updated background sound feature set in each time period to obtain a background sound adaptation sequence.

[0147] Specifically, in this embodiment of the invention, the background sound feature set is first updated according to the update time axis to obtain an updated background sound feature set. Then, the suitability of each sound to the scene is identified through a neural network. For example, if a car horn sounds in a period drama, the suitability of the horn sound is low, while the suitability of wind sounds in a fighting scene is high.

[0148] In this embodiment of the invention, the fit scores of each background sound in the background sound feature set within each time period are arranged to obtain a background sound fit sequence. The background sound fit sequence is represented as [rain sound: 90%, wind sound: 80%, tearing sound: 80%, ... car horn sound: 5%, dog barking sound: 5%].

[0149] S7. The audio data is corrected according to the update timeline to obtain the progress-corrected audio data. The progress-corrected audio data is actively denoised according to the background sound adaptation sequence to obtain the noise-corrected audio data. The noise-corrected audio data and the primary noise-reduced video data are then time-synchronized audio-visual merging operations to obtain the synchronized noise-reduced video.

[0150] In this embodiment of the invention, the audio data is first corrected according to the update timeline to obtain the progress-corrected audio data, and then the progress-corrected audio data is subjected to content-based noise reduction.

[0151] In detail, in this embodiment of the invention, the step of performing active noise reduction on the progress-corrected audio data according to the background sound adaptation sequence to obtain noise-corrected audio data includes:

[0152] Background sound types with fit scores less than a preset noise interference threshold in the background sound fit sequence are extracted to obtain noise background sound types;

[0153] The progress-corrected audio data is subjected to object type identification to obtain an audio object result set;

[0154] The background sounds that match the noise background sound type in the audio object result set are reversed to obtain the reverse audio set;

[0155] Using the inverse audio set, a signal superposition operation is performed on the progress-corrected audio data to obtain noise-corrected audio data.

[0156] In this embodiment of the invention, the noise interference threshold is set to 40%, and background sounds with an adaptation degree of less than 40% can be considered noise by default.

[0157] Then, all noises with low fit in the background sound fit list are identified to obtain the noise background sound type [...car horn 5%, dog barking 5%]. Based on the noise category, active noise reduction is then performed on the progress-corrected audio data to obtain noise-corrected audio data. In this embodiment of the invention, the reverse operation refers to keeping the phase of the audio and noise audio the same, but their magnitudes opposite; therefore, noise can be eliminated through signal superposition.

[0158] Finally, the noise-corrected audio data is combined with the already corrected primary noise-reduced video data to obtain a synchronized noise-reduced video.

[0159] Furthermore, in this embodiment of the invention, after obtaining the synchronized noise-reduced video, the method further includes:

[0160] The background sound fit sequence is visualized to obtain a noise removal list;

[0161] Utilize a pre-built user feedback service to obtain users' noise reduction needs;

[0162] Based on the noise reduction requirements, the background sound adaptation sequence and noise removal list are modified to obtain a synchronized noise-reduced video that meets the user's needs.

[0163] The user feedback service refers to the service where users can directly edit the noise removal list. In this embodiment of the invention, the modification of the background sound adaptation sequence and the noise removal list can directly perform fluctuation filtering and noise reduction on audio and video. The user can be either the author of a film or a viewer.

[0164] To address the problems described in the background section, this invention first performs audio-visual separation to obtain independently processable audio and video data. Then, it preprocesses the audio data and performs time-frequency analysis to obtain a speech feature set and a background noise feature set. The speech feature set can be used to match lip movements in the video data to locate the timeline, while the background noise feature set can analyze noise components for targeted noise reduction. Furthermore, this invention utilizes a pre-trained audio-visual perception noise reduction model to first analyze video features to obtain a video feature set, and then analyze the matching degree between video and audio features. This not only allows for timeline location but also the location of unwanted background noise in the audio, enabling active noise reduction. Finally, the denoised audio and video are merged to obtain a synchronously noise-reduced video. Therefore, this invention can improve the clarity of images and sound in videos.

[0165] like Figure 2 The diagram shown is a functional block diagram of an audio-visual synchronization noise reduction system based on AI visual perception provided in an embodiment of the present invention.

[0166] The AI-based visual perception-based audio-visual synchronization noise reduction system 100 described in this invention can be installed in an electronic device. Depending on the functions implemented, the AI-based visual perception-based audio-visual synchronization noise reduction system 100 may include a raw data acquisition module 101, an audio feature recognition module 102, an audio-visual feature recognition module 103, and a synchronization noise reduction module 104. The module described in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0167] The raw data acquisition module 101 is used to acquire audio and video, perform audio-visual separation operation on the audio and video, and obtain audio data and video data.

[0168] The audio feature recognition module 102 is used to filter the audio data using a pre-built low-pass filter to obtain filtered audio data, and to downsample the filtered audio data according to a preset sampling rate to obtain pre-processed audio data. It also uses a pre-built wavelet transform algorithm to perform time-frequency analysis on the pre-processed audio data to obtain an audio signal feature set, and performs audio separation operation to classify the audio signal feature set into speech audio and background audio to obtain a speech feature set and a background audio feature set.

[0169] The audio-visual feature recognition module 103 is used to perform noise reduction on the video data based on a pre-built optimized minimum mean square error algorithm using a pre-trained audio-visual perception noise reduction model to obtain primary noise-reduced video data, and to perform visual perception operations based on objects and scenes on the primary noise-reduced video data to obtain a video feature set.

[0170] The synchronous noise reduction module 104 is used to perform a timeline correction operation based on lip-shape features on the speech feature set according to the video feature set to obtain an updated timeline; and to perform a progress update operation on the background sound feature set according to the updated timeline to obtain an updated background sound feature set. It then performs a scene adaptation recognition operation on the updated background sound feature set according to the video feature set, outputting the adaptation score of each background sound in the updated background sound feature set within each time period to obtain a background sound adaptation sequence. Furthermore, it performs progress correction on the audio data according to the updated timeline to obtain progress-corrected audio data; performs active noise reduction on the progress-corrected audio data according to the background sound adaptation sequence to obtain noise-corrected audio data; and performs a time-synchronized audio-visual merging operation on the noise-corrected audio data and the primary noise-reduced video data to obtain a synchronous noise-reduced video.

[0171] In detail, the modules in the AI ​​visual perception-based audio-visual synchronization noise reduction system 100 described in this embodiment of the invention employ the same methods as described above. Figure 1 The method used is the same as the AI ​​visual perception-based audio-video synchronization noise reduction method described above, and it can produce the same technical effect, so it will not be repeated here.

[0172] like Figure 3 The diagram shown is a structural schematic of an electronic device that implements an AI-based visual perception-based audio and video synchronization noise reduction method according to an embodiment of the present invention.

[0173] The electronic device 1 may include a processor 10, a memory 11 and a bus 12, and may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as an AI-based visual perception-based audio and video synchronization noise reduction method program.

[0174] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 11 includes both internal and external storage units of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of an AI-based visual perception-based audio-visual synchronization noise reduction method, but also to temporarily store data that has been output or will be output.

[0175] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., an AI-based visual perception-based audio-visual synchronization noise reduction method program), and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0176] The bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 12 can be divided into an address bus, a data bus, a control bus, etc. The bus 12 is configured to realize the connection and communication between the memory 11 and at least one processor 10, etc.

[0177] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0178] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0179] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0180] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0181] The AI-based visual perception-based audio-visual synchronization noise reduction method program stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When run in the processor 10, it can achieve the following:

[0182] Acquire audio and video data, perform audio-visual separation on the audio and video data to obtain audio data and video data;

[0183] The audio data is filtered using a pre-built low-pass filter to obtain filtered audio data, and then downsampled according to a preset sampling rate to obtain preprocessed audio data.

[0184] The preprocessed audio data is subjected to time-frequency analysis using a pre-constructed wavelet transform algorithm to obtain an audio signal feature set. Then, an audio separation operation is performed on the audio signal feature set to classify speech audio and background audio, resulting in a speech feature set and a background audio feature set.

[0185] Using a pre-trained audio-visual perception denoising model, the video data is denoised using a pre-built optimized minimum mean square error algorithm to obtain primary denoised video data. Then, the primary denoised video data is subjected to object and scene-based visual perception operations to obtain a video feature set.

[0186] Based on the video feature set, a timeline correction operation based on lip shape features is performed on the speech feature set to obtain an updated timeline;

[0187] According to the update timeline, a progress update operation is performed on the background sound feature set to obtain an updated background sound feature set. Then, according to the video feature set, a scene adaptability recognition operation is performed on the updated background sound feature set to output the adaptability score of each background sound in the updated background sound feature set in each time period, thus obtaining a background sound adaptability sequence.

[0188] The audio data is progress-corrected according to the update timeline to obtain progress-corrected audio data. Active noise reduction is performed on the progress-corrected audio data according to the background sound adaptation sequence to obtain noise-corrected audio data. The noise-corrected audio data and the primary noise-reduced video data are then time-synchronized audio-visual merging operations to obtain synchronized noise-reduced video.

[0189] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figures 1 to 3 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0190] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0191] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0192] Acquire audio and video data, perform audio-visual separation on the audio and video data to obtain audio data and video data;

[0193] The audio data is filtered using a pre-built low-pass filter to obtain filtered audio data, and then downsampled according to a preset sampling rate to obtain preprocessed audio data.

[0194] The preprocessed audio data is subjected to time-frequency analysis using a pre-constructed wavelet transform algorithm to obtain an audio signal feature set. Then, an audio separation operation is performed on the audio signal feature set to classify speech audio and background audio, resulting in a speech feature set and a background audio feature set.

[0195] Using a pre-trained audio-visual perception denoising model, the video data is denoised using a pre-built optimized minimum mean square error algorithm to obtain primary denoised video data. Then, the primary denoised video data is subjected to object and scene-based visual perception operations to obtain a video feature set.

[0196] Based on the video feature set, a timeline correction operation based on lip shape features is performed on the speech feature set to obtain an updated timeline;

[0197] According to the update timeline, a progress update operation is performed on the background sound feature set to obtain an updated background sound feature set. Then, according to the video feature set, a scene adaptability recognition operation is performed on the updated background sound feature set to output the adaptability score of each background sound in the updated background sound feature set in each time period, thus obtaining a background sound adaptability sequence.

[0198] The audio data is progress-corrected according to the update timeline to obtain progress-corrected audio data. Active noise reduction is performed on the progress-corrected audio data according to the background sound adaptation sequence to obtain noise-corrected audio data. The noise-corrected audio data and the primary noise-reduced video data are then time-synchronized audio-visual merging operations to obtain synchronized noise-reduced video.

[0199] In the embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and actual implementations may have other classification methods.

[0200] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0201] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0202] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0203] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for audio-visual synchronization noise reduction based on AI visual perception, characterized in that, The method includes: Acquire audio and video data, perform audio-visual separation on the audio and video data to obtain audio data and video data; The audio data is filtered using a pre-built low-pass filter to obtain filtered audio data, and then downsampled according to a preset sampling rate to obtain preprocessed audio data. The preprocessed audio data is subjected to time-frequency analysis using a pre-constructed wavelet transform algorithm to obtain an audio signal feature set. Then, an audio separation operation is performed on the audio signal feature set to classify speech audio and background audio, resulting in a speech feature set and a background audio feature set. Using a pre-trained audio-visual perception denoising model, the video data is denoised using a pre-built optimized minimum mean square error algorithm to obtain primary denoised video data. Then, the primary denoised video data is subjected to object and scene-based visual perception operations to obtain a video feature set. Based on the video feature set, a timeline correction operation based on lip shape features is performed on the speech feature set to obtain an updated timeline; According to the update timeline, a progress update operation is performed on the background sound feature set to obtain an updated background sound feature set. Then, according to the video feature set, a scene adaptability recognition operation is performed on the updated background sound feature set to output the adaptability score of each background sound in the updated background sound feature set in each time period, thus obtaining a background sound adaptability sequence. The audio data is corrected according to the update timeline to obtain progress-corrected audio data. Active noise reduction is performed on the progress-corrected audio data according to the background sound adaptation sequence to obtain noise-corrected audio data. The noise-corrected audio data and the primary noise-reduced video data are then time-synchronized audio-visual merging operations to obtain synchronized noise-reduced video. The background sound fit sequence is visualized to obtain a noise removal list; Utilize a pre-built user feedback service to obtain users' noise reduction needs; Based on the noise reduction requirements, the background sound adaptation sequence and noise removal list are modified to obtain a synchronized noise-reduced video that meets the user's needs.

2. The audio-visual synchronization noise reduction method based on AI visual perception as described in claim 1, characterized in that, The pre-constructed wavelet transform algorithm is used to perform time-frequency analysis on the preprocessed audio data to obtain an audio signal feature set, including: Obtain the Haar wavelet, and perform translation and scaling operations on the Haar wavelet to obtain the wavelet function; The preprocessed audio data and the wavelet function are convolved to obtain wavelet coefficients, wherein the wavelet coefficients are expressed as: In the formula, W(a, b) represents wavelet coefficients, a represents scale frequency, b represents position time, t represents time, and x(t) represents the preprocessed audio data. Represents the wavelet function; The wavelet coefficients are reconstructed using a pre-constructed reconstruction formula to obtain an updated wavelet graph, wherein the updated wavelet graph is represented as follows: In the formula, x'(t) represents the updated wavelet graph, j represents the new scale frequency, k represents the new location time, and W j (k) represents the wavelet coefficients at scale frequency j and location time k. Represents the wavelet function at scale frequency j and position time k; The updated wavelet graph is subjected to time- and frequency-based signal feature extraction operations to obtain an audio signal feature set.

3. The audio-visual synchronization noise reduction method based on AI visual perception as described in claim 2, characterized in that, The audio separation operation, which classifies the audio signal feature set into speech audio and background audio, to obtain a speech feature set and a background audio feature set, includes: Perform fully connected audio object recognition on the audio signal feature set to obtain an audio object set; The audio object set is classified to obtain a speech feature set and a background sound feature set. The speech feature set includes the speech of characters in the video, the speech of characters outside the video, and the speech of the video narration. The background sound feature set is the sound type in the audio object set that is not in the speech feature set.

4. The audio-visual synchronization noise reduction method based on AI visual perception as described in claim 3, characterized in that, Prior to utilizing the pre-trained audiovisual perception noise reduction model, the method further includes: Obtain an audiovisual perception denoising model that includes a video perception network, a sound perception network, an active denoising network, and an audio-visual output network; obtain a training sample set, wherein the video perception network and the sound perception network are networks to be trained, the active denoising network and the audio-visual output network are networks with fixed parameters, and the training sample set includes sample audio and video, as well as video object labels, audio object labels and manual correction labels corresponding to the sample audio and video. A target sample is extracted sequentially from the training sample set, and the audio-visual perception noise reduction model is used to perform forward network calculation on the target sample. The pre-built intermediate layer output interface is used to obtain the predicted video object recognition result, the predicted audio object recognition result, and the predicted time axis deviation of the forward network calculation process. Using a pre-built cross-entropy loss algorithm, a first loss value is calculated between the predicted video object recognition result and the video object label; a second loss value is calculated between the predicted audio object recognition result and the audio object label; and a third loss value is calculated between the predicted time axis deviation and the manually corrected label. Based on the pre-built gradient descent algorithm, the first loss value, the second loss value and the third loss value are minimized to obtain a set of sample parameters. The set of sample parameters is then updated in reverse to obtain an updated audio-visual perception noise reduction model. Based on the pre-built model ensemble algorithm, the model parameters of the updated audio-visual perception denoising model corresponding to multiple target samples are obtained, resulting in a set of model parameters. The mean of the set of model parameters is then calculated to obtain the trained audio-visual perception denoising model.

5. The audio-visual synchronization noise reduction method based on AI visual perception as described in claim 4, characterized in that, The step of performing noise reduction on the video data based on a pre-constructed optimized minimum mean square error algorithm to obtain primary noise-reduced video data includes: By obtaining the filter and the minimum mean square error algorithm, the objective function for the minimum mean square error is obtained, wherein the objective function is expressed as: J(w)=E{e 2 [n]}=E{(d[n]-y[n]) 2 } In the formula, J(w) represents the objective function, w represents the coefficients of the filter, E{·} represents the expectation operator, and e 2 [n] represents the square of the error signal, d[n] represents the desired signal of the filter, and y[n] represents the output signal of the filter; Adaptive regularization is introduced into the minimum mean square error algorithm to obtain the optimization objective function, which is expressed as: J'(w)=E{e 2 [n]}+γR(w) In the formula, J'(w) represents the objective function, γ represents the regularization parameter, and R(w) represents the L2 norm regularization term. Represents the square of the Euclidean norm; Obtain the coefficients of the filter, and adjust the coefficients according to the optimization objective function to obtain updated filter coefficients, wherein the updated filter coefficients are expressed as: w[n+1]=w[n]+μe[n]x[n]-2γw[n] In the formula, w[n+1] represents the updated filter coefficients, w[n] represents the initial filter coefficients, μ represents the weight coefficients, and 2γw[n] represents the gradient descent term of the L2 norm regularization term; The video data is filtered according to the updated filtering coefficients to obtain primary noise-reduced video data.

6. The audio-visual synchronization noise reduction method based on AI visual perception as described in claim 5, characterized in that, The process of performing object- and scene-based visual perception operations on the primary denoised video data to obtain a video feature set includes: Perform a convolution operation on the primary noise-reduced video data to obtain a set of video convolution matrices; The set of video convolutional matrices is subjected to dimensionality reduction to obtain a set of video feature sequences; Perform a fully connected recognition operation based on objects and scenes on the video feature sequence set to obtain the video feature set.

7. The audio-visual synchronization noise reduction method based on AI visual perception as described in claim 6, characterized in that, The step of performing a timeline correction operation based on lip-shape features on the speech feature set according to the video feature set to obtain an updated timeline includes: Perform lip-shape feature recognition on the speech feature set to obtain the speech lip-shape recognition result and the time axis corresponding to the speech lip-shape recognition result; Based on the video feature set and the speech lip-sync recognition results, the timeline is subjected to recognition operations based on forward / backward and scaling to obtain the timeline deviation; The time axis is updated based on the time axis deviation to obtain an updated time axis.

8. The audio-visual synchronization noise reduction method based on AI visual perception as described in claim 7, characterized in that, The step of performing active noise reduction on the progress-corrected audio data according to the background sound adaptation sequence to obtain noise-corrected audio data includes: Background sound types with fit scores less than a preset noise interference threshold in the background sound fit sequence are extracted to obtain noise background sound types; The progress-corrected audio data is subjected to object type identification to obtain an audio object result set; The background sounds that match the noise background sound type in the audio object result set are reversed to obtain the reverse audio set; Using the inverse audio set, a signal superposition operation is performed on the progress-corrected audio data to obtain noise-corrected audio data.

9. An AI-based visual perception-based audio-video synchronization noise reduction system, used to execute the AI-based visual perception-based audio-video synchronization noise reduction method of claim 1, characterized in that, The system includes: The raw data acquisition module is used to acquire audio and video, and perform audio-visual separation operation on the audio and video to obtain audio data and video data; The audio feature recognition module is used to filter the audio data using a pre-built low-pass filter to obtain filtered audio data, and to downsample the filtered audio data according to a preset sampling rate to obtain pre-processed audio data. It also uses a pre-built wavelet transform algorithm to perform time-frequency analysis on the pre-processed audio data to obtain an audio signal feature set, and performs audio separation operation to classify the audio signal feature set into speech audio and background audio to obtain a speech feature set and a background audio feature set. The audio-visual feature recognition module is used to perform noise reduction on the video data based on a pre-built optimized minimum mean square error algorithm using a pre-trained audio-visual perception noise reduction model to obtain primary noise-reduced video data, and to perform visual perception operations based on objects and scenes on the primary noise-reduced video data to obtain a video feature set. The synchronous noise reduction module is used to perform a timeline correction operation based on lip-sync features on the speech feature set according to the video feature set to obtain an updated timeline; and to perform a progress update operation on the background sound feature set according to the updated timeline to obtain an updated background sound feature set. It then performs a scene fit recognition operation on the updated background sound feature set according to the video feature set, outputting the fit score of each background sound in the updated background sound feature set within each time period to obtain a background sound fit sequence. Furthermore, it performs progress correction on the audio data according to the updated timeline to obtain progress-corrected audio data; performs active noise reduction on the progress-corrected audio data according to the background sound fit sequence to obtain noise-corrected audio data; and performs a time-synchronized audio-visual merging operation on the noise-corrected audio data and the primary noise-reduced video data to obtain a synchronous noise-reduced video.

Images