Audio denoising method and system based on joint optimization of video understanding and audio recognition

Abstract

This invention discloses an audio denoising method and system based on joint optimization of video understanding and audio recognition, belonging to the field of audio denoising technology. The method includes: acquiring original audio signals and corresponding original video signals within the same time period; performing feature extraction and audio category recognition on the original audio signals to obtain an audio category temporal probability matrix; performing video semantic sound prediction on the original video signals to obtain a video semantic category probability matrix; performing cross-modal temporal alignment between the audio category temporal probability matrix and the video semantic category probability matrix to obtain a cross-modal difference guidance vector; fusing the original audio features, cross-modal difference guidance vector, three-state semantic control vector, and knowledge correction factor through multi-source feature cross-attention to obtain a joint representation; and inputting the joint representation into an audio denoising model to output a preliminary denoising result. This invention improves the semantic accuracy of the denoising result in complex environments.

Description

Technical Field

[0001] This invention relates to the field of audio denoising technology, specifically to an audio denoising method and system based on joint optimization of video understanding and audio recognition. Background Technology

[0002] Existing audio denoising techniques mainly rely on the characteristics of pure audio signals, using spectral analysis or neural network models (such as DNN, UNet, Transformer, etc.) to separate the target sound source from noise in the audio signal.

[0003] However, traditional methods have the following shortcomings: (1) They cannot determine which sounds should appear in the current video scene, so they cannot effectively identify semantically “unreasonable sounds” (such as wind sounds, animal sounds, etc. in indoor scenes) as noise. (2) Models that rely solely on audio features have difficulty distinguishing between semantically similar but different sounds such as “background music” and “useful ambient sounds”.

[0004] However, multimodal information (especially video footage) contains rich semantic context, such as scene category, object type, and environmental features. This information can provide semantic constraints for audio denoising, but it currently lacks systematic utilization.

[0005] In view of the above, this application is hereby submitted. Summary of the Invention

[0006] The technical problem this invention aims to solve is that existing audio denoising methods do not consider multimodal information, especially video image information, resulting in low semantic accuracy. This invention provides an audio denoising method and system based on joint optimization of video understanding and audio recognition. It performs audio denoising based on video prior sound event prediction, cross-modal temporal alignment, and knowledge feedback correction. By extracting scene, object, action, and event information from the video, it predicts the sound category distribution that should exist in the current time period of the video. This sound category distribution is then aligned and compared with the actual sound category distribution detected in the audio, generating a cross-modal difference control signal to guide the audio denoising model to selectively suppress mismatched sound components and retain or compensate for reasonable target sounds, obtaining preliminary denoising results. Simultaneously, a knowledge graph is introduced to verify the logical consistency of the preliminary denoising results and provide feedback correction. This invention improves the semantic accuracy, temporal consistency, logical rationality, and interpretability of the denoising results in complex environments.

[0007] This invention is achieved through the following technical solution:

[0008] In a first aspect, the present invention provides an audio denoising method based on joint optimization of video understanding and audio recognition, the method comprising:

[0009] Acquire the raw audio signal and the corresponding raw video signal within the same time period;

[0010] Feature extraction is performed on the original audio signal to obtain the original audio features; based on the original audio features, multi-label audio category recognition is performed based on the audio multi-category recognition model to obtain the audio category temporal probability matrix;

[0011] Video semantic sound prediction is performed on the original video signal to obtain the video semantic category probability matrix;

[0012] Cross-modal temporal alignment of the audio category probability matrix and the video semantic category probability matrix is ​​performed to obtain the cross-modal difference guide vector;

[0013] The original audio features, cross-modal difference guidance vector, tri-state semantic control vector and knowledge correction factor are fused by multi-source feature cross-attention to obtain a joint representation; the tri-state semantic control vector is obtained by concatenating the retention amount, the anomaly suppression amount and the missing compensation amount.

[0014] The joint representation is input into the audio denoising model, and the initial denoising result is output.

[0015] Further, feature extraction is performed on the original audio signal to obtain the original audio features; based on the original audio features, multi-label audio category recognition is performed based on an audio multi-category recognition model to obtain the audio category temporal probability matrix, including:

[0016] Feature extraction is performed on the original audio signal to obtain the original audio features corresponding to each time frame, and an audio feature sequence is formed.

[0017] The audio feature sequence is input into the audio multi-class recognition model, which outputs the existence probability of each audio class and forms the probability vector of each time frame.

[0018] The probability vectors of each time frame are arranged in chronological order to form an audio category temporal probability matrix; the audio category temporal probability matrix is ​​used to characterize the distribution of multiple types of sound events in the original audio signal at each time frame.

[0019] Furthermore, video semantic sound prediction is performed on the original video signal to obtain the video semantic category probability matrix, including:

[0020] Visual features are obtained by extracting features from the original video signal based on the visual encoder VIT.

[0021] Based on visual features, the video understanding model identifies scenes, objects, actions, and / or events, and forms a structured semantic set.

[0022] Input the structured semantic set into the description generation module to generate semantic text;

[0023] Input the semantic text into the text encoder to obtain the semantic text vector;

[0024] An audio category semantic prototype library is pre-built. The descriptions of each category in the audio category semantic prototype library are input into the same text encoder to obtain category prototype vectors. All category prototype vectors are then used to form the audio category semantic prototype library.

[0025] Calculate the cosine similarity between the semantic text vector and the prototype vector of each category, and normalize all cosine similarities to obtain the video semantic category probability vector for a certain time unit.

[0026] By traversing all time units, a video semantic category probability matrix is ​​formed.

[0027] Furthermore, the expression for the three-state semantic control vector is:

[0028] ;

[0029] ;

[0030] ;

[0031] ;

[0032] In the formula, The retention limit is higher when both audio and video detection determine that the sound should exist.

[0033] The abnormal suppression level is high when the audio detects this type of sound, but the video determines that this type of sound should not exist.

[0034] The compensation amount is higher when the video believes that this type of sound should exist, but the audio detection is weak.

[0035] This represents the probability of the i-th type of sound event occurring in the t-th time frame; Let represent the video semantic category probability vector at time unit t.

[0036] Furthermore, the backbone network of the audio denoising model adopts the Transformer encoder-decoder network architecture, expressed as:

[0037] ;

[0038] ;

[0039] In the formula, The hidden feature representation of the encoder output includes fused global context information and cross-modal semantic information; The encoder function is used to perform multi-layer self-attention computation and feedforward transformation on the input features to extract high-level semantic representations and model long-term temporal dependencies. This is the joint representation obtained by fusing the original audio features, cross-modal difference guidance vector, three-state semantic control vector, and knowledge correction factor through multi-source feature cross-attention fusion; To represent the characteristic representation of the decoder output, the time-frequency mask or frequency band modulation parameters required for audio denoising processing are generated; This represents the decoder function, which is used to further model the target denoising features based on the encoder output and generate a representation for audio reconstruction.

[0040] Furthermore, the method also includes:

[0041] By introducing a knowledge graph, the preliminary denoising results are corrected in a closed loop to obtain the final target denoised frequency signal.

[0042] Furthermore, a knowledge graph is introduced to perform closed-loop correction on the initial denoising results, including:

[0043] The initial denoising results are then input into the audio multi-class recognition model to obtain the re-recognized audio class probability vector.

[0044] The video semantic category probability vector in the video semantic category probability matrix, the re-identified audio category probability vector, and the pre-constructed knowledge graph are input into the knowledge reasoning module to determine whether there is a logical conflict. Logical conflicts include those that should be present but are not, those that should not be present but are, or forced causal omissions.

[0045] If there is a logical conflict, the knowledge correction factor is calculated based on the video semantic category probability vector in the video semantic category probability matrix, the re-identified audio category probability vector, and the pre-constructed knowledge graph.

[0046] The knowledge correction factor is then fed back to the audio denoising model to perform secondary modulation and reconstruction on the local frequency bands, thereby obtaining the final target denoised audio signal.

[0047] Secondly, the present invention provides an audio denoising system based on joint optimization of video understanding and audio recognition, the system comprising:

[0048] The acquisition unit is used to acquire the original audio signal and the corresponding original video signal within the same time period.

[0049] The audio recognition unit is used to extract features from the original audio signal to obtain the original audio features; based on the original audio features, it performs multi-label audio category recognition based on the multi-category audio recognition model to obtain the audio category temporal probability matrix.

[0050] The video semantic prediction unit is used to perform video semantic sound prediction on the original video signal to obtain the video semantic category probability matrix;

[0051] The alignment unit is used to perform cross-modal temporal alignment between the audio category temporal probability matrix and the video semantic category probability matrix to obtain the cross-modal difference guide vector;

[0052] The joint representation unit is used to fuse the original audio features, cross-modal difference guidance vector, tri-state semantic control vector and knowledge correction factor through multi-source feature cross-attention fusion to obtain the joint representation; the tri-state semantic control vector is obtained by concatenating the retention amount, the anomaly suppression amount and the missing compensation amount.

[0053] The audio denoising unit is used to input the joint representation into the audio denoising model and output the preliminary denoising result.

[0054] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned audio denoising method based on joint optimization of video understanding and audio recognition.

[0055] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described audio denoising method based on joint optimization of video understanding and audio recognition.

[0056] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0057] This invention presents an audio denoising method and system based on joint optimization of video understanding and audio recognition. It denoises audio by leveraging prior sound event prediction from the video, cross-modal temporal alignment, and knowledge feedback correction. By extracting scene, object, action, and event information from the video, it predicts the sound category distribution that should exist within the current time period of the video. This sound category distribution is then aligned and compared with the actual sound category distribution detected in the audio, generating a cross-modal difference control signal (i.e., a cross-modal difference guidance vector). This signal guides the audio denoising model to selectively suppress mismatched sound components and retain or compensate for reasonable target sounds, yielding preliminary denoising results. Simultaneously, a knowledge graph is introduced to verify the logical consistency of the preliminary denoising results and provide feedback correction. This invention improves the semantic accuracy, temporal consistency, logical rationality, and interpretability of the denoising results in complex environments. Attached Figure Description

[0058] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0059] Figure 1 The flowchart shows the audio denoising method based on joint optimization of video understanding and audio recognition according to the present invention.

[0060] Figure 2 This is a block diagram of the audio denoising system based on joint optimization of video understanding and audio recognition according to the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0062] The main design features of this invention include:

[0063] 1. Video Semantics-Driven "Present Sound Prior Modeling Mechanism". This invention proposes a mechanism for generating the probability distribution of existing sound categories from video semantics. By analyzing the semantics of scenes, objects, actions, and events in the video and mapping the semantic text to the audio category space, the probability matrix of audio categories that should exist in each time unit is obtained, transforming video information from "auxiliary information" into "denoising decision prior".

[0064] 2. A unified mapping mechanism from text semantics to audio category space. This invention proposes a cross-modal mapping method based on semantic text encoding and audio category semantic prototype matching. By constructing an audio category semantic prototype library and mapping video semantic text to this prototype space, a unified representation of video semantics and audio category space is achieved, providing a foundation for subsequent cross-modal alignment and difference calculation.

[0065] 3. Denoising driven by modeling the difference between expected and actual sound. This invention proposes a cross-modal modeling method based on the difference between the probability distribution of actual audio sound and the probability distribution of expected video sound. By constructing a frame-level difference guidance vector, semantic mismatch information is explicitly introduced into the audio denoising process, enabling the denoising model to distinguish between reasonable and abnormal sounds, thereby improving the semantic accuracy of denoising.

[0066] 4. Three-state semantic control mechanism. This invention proposes a three-state semantic control mechanism, which divides the relationship between the actual audio sound and the video sound that should be stored into three states: retention, anomaly suppression, and missing sound compensation, and generates corresponding control information for each state. This information is used to guide the denoising model to perform differentiated frequency band modulation and mask generation under different semantic conditions.

[0067] 5. Semantic Control to Band-Level Modulation Mapping Mechanism. This invention proposes a mechanism to map category-level semantic control information to band-level dynamic modulation parameters. By converting semantic control vectors into gain coefficients in the frequency dimension, differentiated adjustment of different frequency bands is achieved, enabling semantic priors to directly affect the time-frequency unit-level denoising process.

[0068] 6. Cross-modal temporal alignment and delay robustness mechanism. This invention introduces temporal position coding and local time-shift alignment mechanism in the cross-modal alignment process, enabling audio frames to be weighted and matched with multiple adjacent video frames. This effectively addresses alignment issues caused by inaccurate video semantic prediction timing and sudden audio events, thereby improving the temporal robustness of the system.

[0069] 7. Knowledge Graph-Driven Denoising Closed-Loop Correction Mechanism. This invention introduces a semantic expert verification mechanism based on knowledge graphs. By re-identifying the preliminary denoising results and combining the logical relationships between scene, object, action, and audio category, the consistency of the results is judged. When logical conflicts exist, a correction factor is generated and fed back to the denoising model, thereby achieving closed-loop optimization of the denoising process.

[0070] 8. Interpretable Output Mechanism. This invention achieves an interpretable description of the denoising process by outputting cross-modal difference vectors, semantic control information, and the time intervals corresponding to knowledge correction factors, providing a basis for subsequent audio editing and multimodal analysis.

[0071] Example 1

[0072] like Figure 1 As shown, the present invention provides an audio denoising method based on joint optimization of video understanding and audio recognition, the method comprising:

[0073] S1, acquire the original audio signal within the same time unit. and the corresponding original video signal ;

[0074] S2, for the original audio signal Feature extraction is performed to obtain the original audio features; based on the original audio features, multi-label audio category recognition is performed based on the audio multi-category recognition model to obtain the audio category temporal probability matrix.

[0075] In this embodiment, step S2 specifically includes:

[0076] S21, Based on the local time-frequency feature extraction network, the original audio signal... Feature extraction is performed to obtain the original audio features corresponding to each time frame, and an audio feature sequence is formed. ;

[0077] S22, audio feature sequence In the input audio multi-class recognition model, the output is the existence probability of each audio class, forming a probability vector for each time frame;

[0078] Specifically, the audio multi-class recognition model is a multi-label audio event recognition model, including:

[0079] Temporal context modeling network is used to model the persistence and boundary changes of sound events within multiple adjacent time units;

[0080] The multi-label discrimination layer is used to output the probability of the existence of various sounds within the current time unit.

[0081] Specifically, the local time-frequency feature extraction network can employ a convolutional neural network to obtain:

[0082]

[0083] In the formula, To represent the local time-frequency feature representation of the t-th time frame extracted by the convolutional neural network, used to characterize the structural pattern of the audio signal in the local time and frequency range; To represent the audio time-frequency feature representation corresponding to the t-th time frame, the audio time-frequency feature includes a feature representation obtained based on time-frequency transformation;

[0084] Then, the local features of multiple time units are input into the temporal context modeling network (TCN temporal convolutional neural network, which expands the temporal receptive field through dilated convolutional structures to model long-term temporal dependencies) to obtain:

[0085]

[0086] In the formula, This represents the feature representation of the t-th time unit after introducing global temporal context information. It contains temporal dependency information across time units and is used to characterize the persistence, boundary changes, and multi-class co-occurrence relationships of audio events. This represents the local time-frequency feature representation obtained by the local time-frequency feature extraction network for the i-th time unit, which is used to characterize the local time-frequency structure of the audio within that time unit. This represents a temporal context modeling function, used to perform temporal modeling on multiple time unit features of the input to capture the contextual dependencies of the audio signal in the time dimension. The temporal context modeling function is preferably implemented using a temporal convolutional network (TCN).

[0087] Finally, the presence probability of each audio category is output through a multi-label discrimination layer. :

[0088]

[0089] In the formula, This represents the Sigmoid activation function, which maps the output to the interval [0,1] to represent the probability of each category occurring independently; This represents a learnable weight matrix used to map feature representations to the audio category space; It is the bias vector; = t=1,2,...,T; T is the number of time frames, and each frame t corresponds to the probability of various audio types in the audio. ∈ [0,1], representing the probability of the i-th type of sound event occurring in the t-th time frame.

[0090] S23, Arrange the probability vectors of each time frame in chronological order to form the audio category temporal probability matrix. The audio category temporal probability matrix is ​​used to characterize the distribution of multiple types of sound events in the original audio signal at each time frame.

[0091]

[0092] In the formula, This indicates the total number of predefined audio categories.

[0093] S3, perform video semantic sound prediction on the original video signal to obtain the video semantic category probability matrix;

[0094] In this embodiment, step S3 specifically includes:

[0095] S31, based on the visual encoder VIT, the raw video signal Perform feature extraction to obtain visual features ;

[0096]

[0097] In the formula, This is a visual encoder function used to extract visual features from input video frames or video segments. The visual encoder is preferably implemented using a Vision Transformer (ViT) model. This represents the video frame or video segment corresponding to the t-th time unit;

[0098] S32, based on visual characteristics Based on the video understanding model, it identifies scenes, objects, actions, and / or events, and forms a structured semantic set. ;

[0099]

[0100] In the formula, This represents the scene semantic information identified within the t-th time unit, used to characterize the environment type in which the video is located; This represents the semantic information of the object identified within the t-th time unit, used to characterize the target object appearing in the video; This represents the action semantic information identified within the t-th time unit, used to characterize the behavior or action of the subject in the video; This represents the semantic information of the event identified within the t-th time unit, used to characterize a high-level semantic event consisting of multiple actions or states.

[0101] S33: Input the structured semantic set into the description generation module to generate semantic text. ;

[0102]

[0103] In the formula, This represents the semantic text description corresponding to the t-th time unit, used to characterize the sound-related semantic information that may be present in the video; The description generation module function is used to convert structured semantic information into natural language descriptions. The description generation module can be implemented based on template rules, sequence generation models, or pre-trained language models. This represents the set of structured semantics corresponding to the t-th time unit.

[0104] Semantic text is used to carry semantic information about "what sounds should appear" in a video.

[0105] S34, the semantic text of the t-th time unit. Input text encoder to obtain semantic text vector ;

[0106]

[0107] In the formula, This represents the semantic text vector corresponding to the t-th time unit, used to characterize the potential audio semantic information in the video; This is a text encoder function used to map the input semantic text into a high-dimensional semantic vector. The text encoder can be implemented using one or more of a pre-trained language model, a sentence vector model, or a visual language model. This represents the semantic text description corresponding to the t-th time unit;

[0108] S35, pre-build an audio category semantic prototype library, and set the audio category set. for:

[0109]

[0110] in, The semantic description text is the text corresponding to the i-th audio category. The semantic description text includes one or more of the following: category name, sound source description, typical scene description, or acoustic feature description. Each audio category corresponds to at least one semantic description text.

[0111] By inputting the category descriptions from the audio category semantic prototype library into the same text encoder, category prototype vectors are obtained. ;

[0112]

[0113] In the formula, This is a text encoder function used to map the input semantic description text into a high-dimensional semantic vector; The semantic prototype vector represents the i-th audio category, used to characterize the representation of that audio category in the semantic space.

[0114] The prototype vectors of all categories are used to construct the audio category semantic prototype library. ;

[0115]

[0116] S36, Calculate semantic text vectors With prototype vectors of each category cosine similarity Then, all cosine similarities are normalized to obtain the video semantic category probability vector for the t-th time unit. ;

[0117]

[0118]

[0119] In the formula, This is a temperature parameter used to adjust the smoothness of the similarity distribution;

[0120] Therefore, we get:

[0121]

[0122] Let be the video semantic category probability vector corresponding to the t-th time unit, used to characterize the probability distribution of the existence of each audio category in the video within that time unit;

[0123] S37, Traverse all time units to form the video semantic category probability matrix. .

[0124]

[0125] in, This represents the probability that the i-th type of sound should exist based on the semantics of the video within the t-th time unit. It is not the audio type text itself, but the probability vector obtained by mapping the semantic text to the audio category space.

[0126] S4. Perform cross-modal temporal alignment between the audio category temporal probability matrix and the video semantic category probability matrix to obtain the cross-modal difference guide vector;

[0127] Step S4 takes into account the slight timing offset between video and audio. In this embodiment, a time position code is constructed for each time unit:

[0128]

[0129] Subsequently, a basic difference vector is constructed for each time unit. :

[0130]

[0131] in, This represents the degree of inconsistency between the actual audio sound and the expected video sound across various categories within the t-th time unit.

[0132] Additionally, considering that the sound might arrive later, we can't just look at this one frame; we need to compare it to the preceding and following frames to see which is more similar. We let the audio frame t "reference several nearby video frames," and assign a weighted "trust score" to each frame. Therefore, we use the difference vector as the basis for the difference. Fusion with temporal location encoding: Local time-shifted attention alignment is employed, ensuring that the t-th audio time unit is compared not only with the t-th video time unit, but also with several nearby video time units in a weighted comparison, resulting in:

[0133]

[0134] in:

[0135]

[0136] is the weighted video semantic category probability vector after local time-shift alignment for the t-th time unit;

[0137] This represents the attention weight between the t-th audio time unit and the k-th video time unit, used to characterize the degree of matching between them;

[0138] This represents the normalization function, used to convert similarity values ​​into a probability distribution so that the sum of the weights is 1;

[0139] This represents a query mapping function used to map an audio category probability vector to a query vector.

[0140] This represents a key mapping function used to map video semantic category probability vectors to key vectors;

[0141] This represents the video semantic category probability vector for the k-th time unit;

[0142] This represents the radius of the time window, used to limit the range of neighborhoods participating in the alignment.

[0143] Then use Alternative Participate in the difference calculation to calculate the cross-modal difference steering vector. :

[0144]

[0145] This approach helps improve robustness to slight audiovisual delays.

[0146] S5, the original audio features, cross-modal difference guidance vector, tri-state semantic control vector and knowledge correction factor are fused by multi-source feature cross-attention to obtain a joint representation; the tri-state semantic control vector is obtained by concatenating the retention amount, the anomaly suppression amount and the missing compensation amount.

[0147] In this embodiment, for each category i and each time unit t, the following is defined:

[0148] (1) Quantity to be retained

[0149] When both audio and video detection determine that this type of sound should exist, then the retention amount should be [not specified]. Higher:

[0150]

[0151] in, This represents the probability of the i-th type of sound event occurring in the t-th time frame; Let represent the video semantic category probability vector at time unit t.

[0152] (2) Abnormal inhibition amount

[0153] When the audio detects this type of sound, but the video determines that this type of sound should not exist, the abnormal suppression amount... Higher:

[0154]

[0155] (3) Missing compensation amount

[0156] When the video determines that this type of sound should exist, but the audio detection is weak, then the compensation amount is missing. Higher:

[0157]

[0158] After concatenating the above categories, we obtain the tri-state semantic control vector for the t-th time unit:

[0159]

[0160] The control vector is used to tell the audio denoising model in the subsequent step S6: (1) which sounds should be preserved; (2) which sounds should be suppressed; and (3) which sounds need to be avoided from being suppressed or locally compensated.

[0161] In this embodiment, the original audio features Cross-modal differential guided vector Three-state semantic control vector and knowledge correction factor Multi-source feature cross-attention fusion is performed to obtain a joint representation. The details are as follows:

[0162] Perform linear mapping on various inputs to unify the dimension:

[0163]

[0164]

[0165]

[0166]

[0167] in, , , , They are respectively , , , A corresponding learnable linear mapping parameter matrix is ​​used to map input features from different sources to a feature space of uniform dimension. This parameter matrix consists of learnable linear mapping parameters used to perform dimensionality transformation and feature space alignment on the input features. Through this linear mapping, features from different modalities are aligned within the same semantic space for subsequent fusion processing.

[0168] Then, the above inputs are fused using multi-source feature cross-attention to obtain a joint representation. :

[0169]

[0170] Cross-attention fusion is employed to enhance the targeting of semantic control. The attention model... query key value, and three corresponding biases.

[0171]

[0172]

[0173]

[0174]

[0175]

[0176]

[0177] in, Characterizing the modulation of attention by cross-modal differences; Characterizing the modulation of attention by tri-state semantic control; The modulation of attention by representing knowledge-modified information; For modulation weights;

[0178] The attention weights can then be written as:

[0179]

[0180] The union is represented as:

[0181]

[0182] S6 will be jointly represented Input the audio denoising model and output the preliminary denoising results.

[0183] In this embodiment, the backbone network of the audio denoising model adopts the Transformer encoder-decoder network architecture, expressed as:

[0184]

[0185]

[0186] In the formula, The hidden feature representation of the encoder output includes fused global context information and cross-modal semantic information; The encoder function is used to perform multi-layer self-attention computation and feedforward transformation on the input features to extract high-level semantic representations and model long-term temporal dependencies. This is the joint representation obtained by fusing the original audio features, cross-modal difference guidance vector, three-state semantic control vector, and knowledge correction factor through multi-source feature cross-attention fusion; To represent the characteristic representation of the decoder output, the time-frequency mask or frequency band modulation parameters required for audio denoising processing are generated; This represents the decoder function, which is used to further model the target denoising features based on the encoder output and generate a representation for audio reconstruction.

[0187] To implement category-level control signals to specific frequency regions, this embodiment also includes a dynamic frequency band modulation layer, which generates frequency band gain based on semantic control and knowledge correction information.

[0188]

[0189] in, This represents the gain control vector for each frequency bin; The activation function is used to perform a non-linear mapping on the output, preferably the Sigmoid function, to limit the gain within a preset range; This represents a learnable linear mapping parameter matrix used to map semantic control information to the frequency dimension; This indicates the bias term.

[0190] The original time-frequency characteristics are modulated using this gain:

[0191]

[0192] in, This indicates the time-frequency characteristics of the audio after band modulation; These are the original audio features.

[0193] Then, the time-frequency mask is output by the denoising backbone network. :

[0194]

[0195] in, This represents a time-frequency mask, used to selectively preserve or suppress audio signals at each time-frequency unit. The activation function, preferably the Sigmoid function, is used to constrain the mask value to the interval [0,1]. is a learnable linear mapping parameter matrix; This is the feature representation output by the decoder, which contains control information for audio denoising; For bias terms;

[0196] The time-frequency mask is applied to the modulated time-frequency features. :

[0197]

[0198] Finally, the waveform is reconstructed using the inverse short-time Fourier transform:

[0199]

[0200] By concatenating all time units, a preliminary denoising result is obtained. .

[0201] As a further implementation, the method also includes:

[0202] S7 introduces a knowledge graph to perform closed-loop correction on the preliminary denoising results, and obtains the final target denoised frequency signal.

[0203] Step S7 above takes into account that the initial denoising results may still have semantic inconsistencies. Therefore, a knowledge graph is introduced to perform closed-loop correction on the initial denoising results.

[0204] In detail, step S7 includes:

[0205] S71, the preliminary denoising result is input again into the audio multi-class recognition model to obtain the re-recognized audio class probability vector. ;

[0206]

[0207] S72, the video semantic category probability vector in the video semantic category probability matrix. The re-identified audio category probability vector The pre-built knowledge graph (KG) is input into the knowledge reasoning module to determine whether there are logical conflicts. Logical conflicts include those that should be present but are not, those that should not be present but are, or forced causal omissions.

[0208] Specifically, a knowledge graph (KG) includes at least:

[0209] a. Scene-sound co-occurrence relationship;

[0210] b. Strong causal relationship between action and sound;

[0211] c. The potential relationship between objects and sound;

[0212] d. Scene-sound conflict relationship;

[0213] e. Event-sound continuity pattern relationship.

[0214] For example:

[0215] The office setting is filled with the sounds of keyboards, people talking, and low-level noise from the air conditioner.

[0216] There is a causal relationship between the drumming action and the intensity of the drum sound;

[0217] The potential connection between guitar objects and guitar sounds;

[0218] The office setting clashes with the sound of ocean waves.

[0219] S73. If a logical conflict exists, the knowledge correction factor is calculated based on the video semantic category probability vector in the video semantic category probability matrix, the re-identified audio category probability vector, and the pre-constructed knowledge graph. ;

[0220]

[0221] The Reason() module is a knowledge graph-based semantic reasoning model. It outputs correction values ​​for each category by jointly analyzing audio categories, video semantics, and semantic relationships within the knowledge graph. The Reason() module is implemented using a graph neural network, modeling category relationships within the knowledge graph to achieve semantic reasoning across multiple categories. Specifically, audio categories are used as graph nodes to construct a knowledge graph-based relationship graph structure, where nodes represent audio categories and edges represent semantic relationships between categories, including co-occurrence, causal, and conflict relationships.

[0222] For each category node, construct the initial feature representation. :

[0223]

[0224] in: This represents the probability of the video being predicted for that category. This indicates the probability of the audio being classified into that category. This represents the semantic embedding representation of the category in the knowledge graph.

[0225] Node features are propagated and updated using graph neural networks:

[0226]

[0227] Finally, the knowledge correction factors for each category are obtained through the output layer:

[0228]

[0229] And constitute:

[0230]

[0231] like This indicates that the corresponding category is being enhanced or protected.

[0232] like This indicates further suppression of the corresponding category.

[0233] S74, and feeds back the knowledge correction factor to the audio denoising model, performs secondary modulation and reconstruction on the local frequency band, and obtains the final target denoised frequency signal.

[0234] Example 2

[0235] like Figure 2 As shown, the difference between this embodiment and Embodiment 1 is that this embodiment provides an audio denoising system based on joint optimization of video understanding and audio recognition. This system corresponds one-to-one with the audio denoising method based on joint optimization of video understanding and audio recognition in Embodiment 1. The system includes:

[0236] The acquisition unit is used to acquire the original audio signal and the corresponding original video signal within the same time period.

[0237] The audio recognition unit is used to extract features from the original audio signal to obtain the original audio features; based on the original audio features, it performs multi-label audio category recognition based on the multi-category audio recognition model to obtain the audio category temporal probability matrix.

[0238] The video semantic prediction unit is used to perform video semantic sound prediction on the original video signal to obtain the video semantic category probability matrix;

[0239] The alignment unit is used to perform cross-modal temporal alignment between the audio category temporal probability matrix and the video semantic category probability matrix to obtain the cross-modal difference guide vector;

[0240] The joint representation unit is used to fuse the original audio features, cross-modal difference guidance vector, tri-state semantic control vector and knowledge correction factor through multi-source feature cross-attention fusion to obtain the joint representation; the tri-state semantic control vector is obtained by concatenating the retention amount, the anomaly suppression amount and the missing compensation amount.

[0241] The audio denoising unit is used to input the joint representation into the audio denoising model and output the preliminary denoising result.

[0242] The execution process of each unit can be carried out according to the audio denoising method based on joint optimization of video understanding and audio recognition in Example 1, and will not be described in detail in this example.

[0243] Meanwhile, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-mentioned audio denoising method based on joint optimization of video understanding and audio recognition.

[0244] Meanwhile, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described audio denoising method based on joint optimization of video understanding and audio recognition.

[0245] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0246] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0247] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0248] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0249] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An audio denoising method based on joint optimization of video understanding and audio recognition, characterized in that, The method includes: Acquire the raw audio signal and the corresponding raw video signal within the same time period; Feature extraction is performed on the original audio signal to obtain the original audio features; based on the original audio features, multi-label audio category recognition is performed based on an audio multi-category recognition model to obtain the audio category temporal probability matrix. Perform video semantic sound prediction on the original video signal to obtain a video semantic category probability matrix; The audio category temporal probability matrix and the video semantic category probability matrix are aligned across modal temporally to obtain a cross-modal difference guide vector; The original audio features, cross-modal difference guidance vector, tri-state semantic control vector, and knowledge correction factor are fused by multi-source feature cross-attention to obtain a joint representation; the tri-state semantic control vector is obtained by concatenating the retention amount, the anomaly suppression amount, and the missing compensation amount. The joint representation is input into the audio denoising model, and the preliminary denoising result is output.

2. The audio denoising method based on joint optimization of video understanding and audio recognition according to claim 1, characterized in that, Feature extraction is performed on the original audio signal to obtain the original audio features; Based on the original audio features, multi-label audio category recognition is performed using an audio multi-category recognition model to obtain an audio category temporal probability matrix, including: Feature extraction is performed on the original audio signal to obtain the original audio features corresponding to each time frame, and an audio feature sequence is formed; The audio feature sequence is input into the audio multi-class recognition model, which outputs the existence probability of each audio class and forms a probability vector for each time frame. The probability vectors of each time frame are arranged in chronological order to form an audio category temporal probability matrix; the audio category temporal probability matrix is ​​used to characterize the distribution of multiple types of sound events in the original audio signal at each time frame.

3. The audio denoising method based on joint optimization of video understanding and audio recognition according to claim 1, characterized in that, Perform video semantic sound prediction on the original video signal to obtain a video semantic category probability matrix, including: Visual features are extracted from the original video signal based on the visual encoder VIT to obtain visual features. Based on the visual features, the video understanding model identifies the scene, object, action and / or event respectively, and forms a structured semantic set; The structured semantic set is input into the description generation module to generate semantic text; The semantic text is input into a text encoder to obtain a semantic text vector; An audio category semantic prototype library is pre-built. The descriptions of each category in the audio category semantic prototype library are input into the same text encoder to obtain category prototype vectors. All category prototype vectors are then used to form the audio category semantic prototype library. Calculate the cosine similarity between the semantic text vector and the prototype vector of each category, and normalize all cosine similarities to obtain the video semantic category probability vector for a certain time unit. By traversing all time units, a video semantic category probability matrix is ​​formed.

4. The audio denoising method based on joint optimization of video understanding and audio recognition according to claim 1, characterized in that, The expression for the tri-state semantic control vector is: ； ； ； ； In the formula, The amount to be retained; This represents an abnormal inhibition level; This is the amount of compensation for the missing information; This represents the probability of the i-th type of sound event occurring in the t-th time frame; Let represent the video semantic category probability vector at time unit t.

5. The audio denoising method based on joint optimization of video understanding and audio recognition according to claim 1, characterized in that, The backbone network of the audio denoising model adopts the Transformer encoder-decoder network architecture, and its expression is: ； ； In the formula, The hidden feature representation of the encoder output includes fused global context information and cross-modal semantic information; The encoder function is used to perform multi-layer self-attention computation and feedforward transformation on the input features to extract high-level semantic representations and model long-term temporal dependencies. This is the joint representation obtained by fusing the original audio features, cross-modal difference guidance vector, three-state semantic control vector, and knowledge correction factor through multi-source feature cross-attention fusion; To represent the characteristic representation of the decoder output, the time-frequency mask or frequency band modulation parameters required for audio denoising processing are generated; This represents the decoder function, which is used to further model the target denoising features based on the encoder output and generate a representation for audio reconstruction.

6. The audio denoising method based on joint optimization of video understanding and audio recognition according to claim 1, characterized in that, The method also includes: By introducing a knowledge graph, the preliminary denoising results are corrected in a closed loop to obtain the final target denoised frequency signal.

7. The audio denoising method based on joint optimization of video understanding and audio recognition according to claim 6, characterized in that, A knowledge graph is introduced to perform closed-loop correction on the preliminary denoising results, including: The initial denoising result is input again into the audio multi-class recognition model to obtain the re-recognized audio class probability vector; The video semantic category probability vector in the video semantic category probability matrix, the re-identified audio category probability vector, and the pre-constructed knowledge graph are input into the knowledge reasoning module to determine whether there is a logical conflict; the logical conflict includes something that should be there but is not, something that should not be there but is there, or a forced causal absence. If a logical conflict exists, a knowledge correction factor is calculated based on the video semantic category probability vector in the video semantic category probability matrix, the re-identified audio category probability vector, and the pre-constructed knowledge graph. The knowledge correction factor is then fed back to the audio denoising model to perform secondary modulation and reconstruction on the local frequency bands, thereby obtaining the final target denoised audio signal.

8. An audio denoising system based on joint optimization of video understanding and audio recognition, characterized in that, The system includes: The acquisition unit is used to acquire the original audio signal and the corresponding original video signal within the same time period. An audio recognition unit is used to extract features from the original audio signal to obtain original audio features; and based on the original audio features, to perform multi-label audio category recognition based on an audio multi-category recognition model to obtain an audio category temporal probability matrix. The video semantic prediction unit is used to perform video semantic sound prediction on the original video signal to obtain a video semantic category probability matrix; An alignment unit is used to perform cross-modal temporal alignment between the audio category temporal probability matrix and the video semantic category probability matrix to obtain a cross-modal difference guide vector; The joint representation unit is used to perform multi-source feature cross-attention fusion of the original audio features, cross-modal difference guidance vector, tri-state semantic control vector and knowledge correction factor to obtain the joint representation; the tri-state semantic control vector is obtained by concatenating the retention amount, the anomaly suppression amount and the missing compensation amount. An audio denoising unit is used to input the joint representation into an audio denoising model and output a preliminary denoising result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the audio denoising method based on joint optimization of video understanding and audio recognition as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the audio denoising method based on joint optimization of video understanding and audio recognition as described in any one of claims 1 to 7.

Images