A method and system for separating audio from a specific person in a multi-source environment

Abstract

This invention discloses a method and system for separating audio from a specific person in a multi-sound-source environment, belonging to the field of artificial intelligence and multimodal signal processing technology. The method includes acquiring a visual image sequence of the target object and extracting corresponding visual features; acquiring a noisy frequency signal within the region and extracting corresponding acoustic features; and fusing the visual and acoustic features using a cross-modal correlation mechanism to reconstruct the target audio. Based on acquiring a visual image sequence of the target object and extracting corresponding visual features; acquiring a noisy frequency signal within the region and extracting corresponding acoustic features; and fusing the visual and acoustic features using a cross-modal correlation mechanism to reconstruct the target audio, this method can reasonably distinguish between the same visual and sound sources, facilitating subsequent processing and helping to solve existing technical problems.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimodal signal processing technology, and in particular to a method and system for separating the audio of a specific person in a multi-source environment. Background Technology

[0002] Some social events involve different types and multiple sound sources, forming complex acoustic scenarios. In practice, it is sometimes necessary to separate the pure speech of the target speaker from a signal mixed with background noise or interference from other speakers.

[0003] Existing technologies face three major pain points in commercialization:

[0004] 1. Failure in Dynamic Noise Environments: In robotic scenarios, the motor and fan noise generated by the robot's own movement constitutes near-field broadband noise, which is difficult to eliminate using traditional pure audio noise reduction methods. In security scenarios, bank branches or government service halls are noisy, and surveillance videos often "show the image clearly but cannot hear the conversation," leading to difficulties in determining liability in business disputes.

[0005] 2. The conflict between edge computing power and privacy: Existing high-performance audiovisual models require enormous computational resources, making it difficult to run on edge devices with limited computing power (such as surveillance NVRs, which typically have less than 10 TOPS of computing power). Uploading to the cloud for processing, however, poses extremely high data privacy compliance risks in financial and security scenarios.

[0006] 3. Existing technical limitations: Existing pure audio solutions perform extremely poorly in low signal-to-noise ratio environments, making it difficult to achieve commercial usability. Summary of the Invention

[0007] This invention proposes a method and system for separating the audio of a specific person in a multi-source environment to address the various shortcomings of existing technologies.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A method for separating audio of a specific person in a multi-source environment includes: acquiring a visual image sequence of the target object and extracting the corresponding visual features; acquiring a noisy frequency signal in the region and extracting the corresponding acoustic features; and fusing the visual features and the acoustic features using a cross-modal correlation mechanism to reconstruct the target audio.

[0010] Furthermore, this method is applicable to edge computing devices; correspondingly, the acquisition of noisy frequency signals within the region and the extraction of corresponding acoustic features includes: processing the noisy frequency signals within the region based on a temporal compression coding strategy to extract the acoustic features, wherein the stride of the convolutional layer is set to an integer greater than or equal to 2 to reduce the temporal dimension length of the feature sequence and adapt to the storage resources of the edge computing device.

[0011] Furthermore, the stride of the convolutional layer is set to 4.

[0012] Furthermore, the acquisition of the noisy frequency signal within the region and the extraction of the corresponding acoustic features include: forcibly resampling the original noisy frequency signal to 16kHz; dividing the resampled audio into frames, with each segment having an inference duration of 4 seconds; and normalizing the waveform so that its values ​​are distributed between [-1, 1].

[0013] Furthermore, the acquisition of the visual image sequence of the target object and the extraction of corresponding visual features include: identifying the target's lip movements and corresponding lip temporal sequences. This process includes: locating the lip region using a face detection model; cropping a Region of Interest (ROI) with the center of the lips as the origin, with a fixed size of 96×96 pixels; uniformly interpolating and adjusting the video frame rate to 25fps; and inputting a visual tensor. ,in 100 frames.

[0014] Furthermore, the method of fusing the visual features and the acoustic features using a cross-modal correlation mechanism includes: generating an attention mask to suppress noise components in the acoustic features that are inconsistent with the lip timing.

[0015] Furthermore, the method of fusing the visual and acoustic features using a cross-modal correlation mechanism to reconstruct the target audio includes: constructing and training a dedicated multimodal neural network model, comprising three sub-networks: an audio encoder, a visual encoder, and a cross-modal fusion module; the corresponding layer parameter configuration includes: the audio encoder comprising: an input layer, which is a one-dimensional convolutional layer with 1 input channel, 64 output channels, 8 kernels, a stride of 4, and padding of 2; and a deep feature extraction network, comprising four sequentially connected residual one-dimensional convolutional blocks for outputting the extracted acoustic features. The visual encoder includes: a front-end network composed of three-dimensional convolutional layers; a backbone network, which is a deep residual convolutional network with fully connected layers removed; and a temporal alignment module, used to upsample the temporal dimension of the extracted visual features using a nearest neighbor interpolation algorithm to obtain the temporal dimension of the acoustic features. Consistent alignment of visual features across time dimensions The cross-modal fusion module is used to generate fusion features through a corresponding mechanism: obtaining the query vector Q, key vector K, and value vector V through a mapping layer, wherein... The The above The above These represent the corresponding learnable weight matrices; the attention feature map is calculated using the attention mechanism:

[0016] Where dk is the dimension of the key vector K, This represents the matrix transpose operation; the final output fused features are obtained through a gated fusion mechanism. :

[0017] ;in, Let be the weight matrix of the gating module, sigmoid be the activation function, and ⊙ represent the element-wise multiplication operation of the matrix.

[0018] Furthermore, the training process of the dedicated neural network model includes: constructing a loss function based on the scale-invariant signal-to-noise ratio, the corresponding formula of which includes:

[0019] ;

[0020] Where starget represents the projection component of the target clean speech signal onto the estimated speech signal, and enoise represents the residual noise component in the estimated speech signal; the dedicated neural network model is iteratively trained using a sample set containing noisy frequencies from multiple sources and their corresponding clean target audio, until the loss function converges on the validation set.

[0021] Furthermore, this method is applicable to financial institutions equipped with edge computing devices; correspondingly, the area is the counter area, and the target objects are tellers and customers; the method also includes: inputting the reconstructed target audio into a locally deployed automatic speech recognition model for processing.

[0022] A system for separating the audio of a specific person in a multi-source environment includes:

[0023] The first module is used to acquire the visual image sequence of the target object and extract the corresponding visual features;

[0024] The second module is used to acquire noisy frequency signals within the region and extract the corresponding acoustic features;

[0025] The third module is used to fuse the visual features and the acoustic features using a cross-modal correlation mechanism to reconstruct the target audio.

[0026] By adopting the above technical solution, the present invention has the following beneficial effects:

[0027] 1. This invention is based on acquiring a visual image sequence of a target object and extracting the corresponding visual features; acquiring a noisy frequency signal within a region and extracting the corresponding acoustic features; and using a cross-modal correlation mechanism to fuse the visual features and the acoustic features to reconstruct the target audio. This can reasonably distinguish between the same visual and sound sources, facilitating various subsequent processing and helping to solve existing technical problems. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of a method for separating the audio of a specific person in a multi-sound-source environment proposed in this invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] like Figure 1 The method for separating audio of a specific person in a multi-source environment includes: S1, acquiring a visual image sequence of the target object and extracting the corresponding visual features; S2, acquiring a noisy frequency signal in the region and extracting the corresponding acoustic features; S3, fusing the visual features and the acoustic features using a cross-modal correlation mechanism to reconstruct the target audio.

[0031] Data preprocessing and input construction:

[0032] 1. Audio Stream Processing: Force resampling of the original audio to 16kHz. Divide the audio into frames, setting the inference duration of each segment to 4 seconds. Normalize the waveform so that its values ​​are distributed between [-1, 1].

[0033] 2. Visual stream processing: Use a face detection model to locate the lip region and crop a ROI region with a fixed size of 96×96 pixels. The video frame rate is then uniformly interpolated and adjusted to 25fps.

[0034] Detailed parameters of the neural network model: The multimodal model of this invention contains three sub-networks:

[0035] 1. Audio Encoder (Edge-Adapted Version): One-dimensional convolutional layers have 1 input channel and 64 output channels. Core parameters include kernel size of 8, stride of 4, and padding of 2. Downsampling with a stride of 4 significantly reduces GPU memory usage. Deep feature extraction connects to 4 residual one-dimensional convolutional blocks.

[0036] 2. Visual encoder: The front end consists of a single 3D convolutional layer; the backbone uses a deep residual convolutional network (with the last fully connected layer removed); temporal alignment uses nearest neighbor interpolation to upsample the temporal dimension of visual features.

[0037] 3. Cross-modal fusion module and decoding: The query vector Q, key vector K, and value vector V are obtained through a mapping layer. Attention mechanisms and gating fusion are used to generate output features. The decoder structure is symmetrical to the encoder, and the output layer is a one-dimensional transposed convolutional layer.

[0038] Model training process: Scale-invariant signal-to-noise ratio loss is used. The loss function is used as the validation set. Training is performed using a sample set containing noisy frequencies from multiple sources until the loss function converges on the validation set.

[0039] Commercial Scenario Example (Edge-Based Intelligent Evidence Collection System for Bank Counters): An edge computing box (such as an NPU device with 8 TOPS of computing power) is deployed in the branch's computer room, connecting to the counter's high-definition camera and microphone. The system simultaneously locks onto the lips of both the teller and the customer, using a local, large-step, lightweight model to separate the "teller audio track" and "customer audio track" from a single-channel mixed recording in real time, and sends the clean speech to the local automatic speech recognition (ASR) engine. The raw data is not uploaded to the cloud, meeting financial compliance requirements.

[0040] Applicable to edge computing devices; correspondingly, the acquisition of noisy frequency signals within the region and extraction of corresponding acoustic features includes: processing the noisy frequency signals within the region based on a temporal compression coding strategy to extract the acoustic features, wherein the stride of the convolutional layer is set to an integer greater than or equal to 2 to reduce the temporal dimension length of the feature sequence and adapt to the storage resources of the edge computing device.

[0041] The stride of the convolutional layer is set to 4.

[0042] The acquisition of the noisy frequency signal within the region and the extraction of the corresponding acoustic features include: forcibly resampling the original noisy frequency signal to 16kHz; dividing the resampled audio into frames, with each segment having an inference duration of 4 seconds; and normalizing the waveform so that its values ​​are distributed between [-1, 1].

[0043] The process of acquiring the visual image sequence of the target object and extracting the corresponding visual features includes: recognizing the lip movements and corresponding lip temporal sequences of the target. This process includes: locating the lip region using a face detection model; cropping a Region of Interest (ROI) with the center of the lips as the origin, with a fixed size of 96×96 pixels; uniformly interpolating and adjusting the video frame rate to 25fps; and inputting a visual tensor. ,in 100 frames.

[0044] The method of fusing the visual features and the acoustic features using a cross-modal correlation mechanism includes: generating an attention mask to suppress noise components in the acoustic features that are inconsistent with the lip timing.

[0045] The method of fusing visual and acoustic features using a cross-modal correlation mechanism to reconstruct the target audio includes: constructing and training a dedicated multimodal neural network model, comprising three sub-networks: an audio encoder, a visual encoder, and a cross-modal fusion module; the corresponding layer parameter configuration includes: the audio encoder comprising: an input layer, which is a one-dimensional convolutional layer with 1 input channel, 64 output channels, 8 kernels, a stride of 4, and padding of 2; and a deep feature extraction network comprising four sequentially connected residual one-dimensional convolutional blocks for outputting the extracted acoustic features. The visual encoder includes: a front-end network composed of three-dimensional convolutional layers; a backbone network, which is a deep residual convolutional network with fully connected layers removed; and a temporal alignment module, used to upsample the temporal dimension of the extracted visual features using a nearest neighbor interpolation algorithm to obtain the temporal dimension of the acoustic features. Consistent alignment of visual features across time dimensions The cross-modal fusion module is used to generate fusion features through a corresponding mechanism: obtaining the query vector Q, key vector K, and value vector V through a mapping layer, wherein... The The above The above These represent the corresponding learnable weight matrices; the attention feature map is calculated using the attention mechanism:

[0046] ;in, Let K be the dimension of the key vector. This represents the matrix transpose operation; the final output fused features are obtained through a gated fusion mechanism. :

[0047] ;in, Let be the weight matrix of the gating module, sigmoid be the activation function, and ⊙ represent the element-wise multiplication operation of the matrix.

[0048] The training process of the dedicated neural network model includes: constructing a loss function based on the scale-invariant signal-to-noise ratio, the corresponding formula of which includes:

[0049] ;

[0050] Where starget represents the projection component of the target clean speech signal onto the estimated speech signal, and enoise represents the residual noise component in the estimated speech signal; the dedicated neural network model is iteratively trained using a sample set containing noisy frequencies from multiple sources and their corresponding clean target audio, until the loss function converges on the validation set.

[0051] This method is applicable to financial institutions equipped with edge computing devices; the corresponding area is the counter area, and the target objects are tellers and customers; the method further includes: inputting the reconstructed target audio into a locally deployed automatic speech recognition model for processing.

[0052] A system for separating the audio of a specific person in a multi-source environment includes:

[0053] The first module is used to acquire the visual image sequence of the target object and extract the corresponding visual features;

[0054] The second module is used to acquire noisy frequency signals within the region and extract the corresponding acoustic features;

[0055] The third module is used to fuse the visual features and the acoustic features using a cross-modal correlation mechanism to reconstruct the target audio.

[0056] This invention aims to solve the above-mentioned problems and provide a lightweight, high-precision audiovisual fusion speech enhancement solution. The core innovations include:

[0057] Edge-Friendly Architecture: This innovative architecture employs a large stride (≥ 4) temporal compression coding strategy. Compared to traditional schemes with stride=1 or 2, this invention compresses the length of the audio feature sequence to 1 / 4 of its original length, significantly reducing memory footprint and computational load, and resolving the memory overflow (OOM) problem when running high-precision models on edge chips.

[0058] Visual-Guided Active Hearing: This method uses a camera to capture lip movements and generates an "auditory mask" through a cross-modal attention mechanism. This forces the model to focus only on the sound frequency band that is synchronized with the lip movements, effectively blocking out environmental noise and non-target human voices.

[0059] Privacy-preserving computing deployment solution: It achieves complete localization of audio-visual separation, ensuring that raw biometric data does not leave the local area network / device, thus meeting the high-level compliance requirements of financial security.

[0060] Data preprocessing and input construction

[0061] To ensure the alignment and normalization of multimodal data, this embodiment uses the following specific parameters:

[0062] 1. Audio stream processing:

[0063] Use librosa or ffmpeg to force a resample of the original audio to 16kHz (Fs = 16000).

[0064] The audio is divided into frames, and the inference duration of each frame is set to 4 seconds, i.e., the input audio vector. .

[0065] The waveform is normalized so that its values ​​are distributed between [−1, 1].

[0066] 2. Visual Stream Processing:

[0067] Use a face detection model (such as dlib or RetinaFace) to locate the lip region.

[0068] Extract the ROI region with the center of the lips as the origin, and fix the size to 96 × 96 pixels.

[0069] The video frame rate was uniformly interpolated and adjusted to 25fps.

[0070] Input visual tensor ,in 100 frames (corresponding to 4 seconds of audio).

[0071] Detailed parameters of neural network model

[0072] The AttentiveAV model of this invention specifically includes the following three sub-networks, with the following hierarchical parameter configurations:

[0073] Audio Encoder - Edge-Adapted Version

[0074] Input layer: One-dimensional convolutional layer (Conv1d);

[0075] Input channel: 1; Output channel: 64;

[0076] Key innovative parameters: Kernel Size = 8, Stride = 4, Padding = 2.

[0077] Implementation results: By downsampling with Stride=4, 64,000 sampling points were compressed into 16,000 feature points, reducing the feature length by 75% and significantly reducing memory usage.

[0078] Deep feature extraction:

[0079] Connect 4 layers of residual one-dimensional convolutional blocks (ResBlock-1D).

[0080] Output: Acoustic features .

[0081] Visual Encoder

[0082] Front-end: Consists of a single 3D convolutional layer (Conv3d);

[0083] Backbone: ResNet-18 network (with the last fully connected layer removed);

[0084] Temporal alignment: Nearest neighbor interpolation is used to upsample the temporal dimension of visual features to make it consistent with the temporal dimension of audio features.

[0085] Cross-modal fusion module - attention mechanism

[0086] Mapping layer: ;

[0087] Where Q is the query vector, K is the key vector, and V is the value vector.

[0088] Audio Decoder

[0089] The structure is symmetrical to the encoder.

[0090] Output layer: One-dimensional transposed convolutional layer (ConvTranspose1d).

[0091] Key parameters: kernel size = 8, stride = 4.

[0092] Activation function: Tanh, which limits the output amplitude to [−1, 1].

[0093] The specific implementation of the neural network training process:

[0094] Loss Function: Scale-invariant signal-to-noise ratio loss is used. Loss), the formula is as follows:

[0095] , where starget is the projection of the target clean speech onto the estimated speech, and enoise is the residual noise.

[0096] Convergence criterion: Train on the LRS2 dataset for approximately 20 rounds until the validation set loss no longer decreases significantly.

[0097] Commercial Scenario Examples

[0098] Example A: Intelligent Evidence Collection System at the Edge of Bank Counter

[0099] Deployment architecture: Deploy an edge computing box (such as an NPU box with 8 TOPS computing power) in the branch office's computer room, and connect it to the counter's high-definition anti-riot camera (to acquire visual stream) and microphone (to acquire audio stream).

[0100] Business logic:

[0101] The system uses the algorithm described above to simultaneously lock onto the lips of both the teller and the customer.

[0102] Using a lightweight model with Stride=4, the "teller audio track" and "customer audio track" can be separated locally in real time from a single-channel mixed recording.

[0103] The separated pure voice and video are synthesized and stored, and the original data is not uploaded to the cloud, which meets financial compliance requirements.

[0104] Example B: The noise-resistant interactive service robot's camera locks onto the user's lips and generates an "auditory mask" through a cross-modal attention mechanism. It actively filters out background noise and its own motor noise other than the user's voice and sends the clean voice to the local ASR recognition command.

[0105] Beneficial effects

[0106] Performance Breakthrough: In practical tests on the LRS2 dataset, this invention will... The improvement from -0.17 dB to 0.94 dB represents a qualitative leap from "unusable" to "usable".

[0107] Extreme optimization of resource efficiency: By downsampling optimization with Stride=4, memory usage is reduced by about 60% compared to traditional methods, making it possible to train and deploy long sequence audiovisual models on edge devices with 16GB or even less memory on a single card.

[0108] Highly commercially viable: The technical solution fully considers financial compliance (localized processing) and hardware costs (low computing power adaptation), making it highly valuable for promotion.

[0109] The beneficial effects of this invention include:

[0110] 1. Edge-friendly temporal compression architecture: It innovatively adopts a large stride (Stride ≥ 4) temporal compression coding strategy to compress the length of audio feature sequences to 1 / 4 of the original length, which significantly reduces memory usage and computational load, and solves the memory overflow problem when running high-precision models on edge chips.

[0111] 2. Visually Guided Active Hearing: By using a camera to capture lip movements, a cross-modal attention mechanism forces the model to focus only on the sound frequency band synchronized with the lip movements, effectively blocking out environmental noise.

[0112] 3. Privacy Computing Deployment Solution: It achieves complete localization of audio-visual separation, ensuring that raw biometric data does not leave the local area network, thus meeting the high-level compliance requirements of financial security.

[0113] The above description is a detailed description of the preferred embodiments of the present invention. However, the embodiments are not intended to limit the scope of the patent application of the present invention. All equivalent changes or modifications made under the technical spirit of the present invention should fall within the patent scope covered by the present invention.

Claims

1. A method for separating audio from a specific person in a multi-source environment, characterized in that, include: Obtain the visual image sequence of the target object and extract the corresponding visual features; Acquire noisy frequency signals within the region and extract the corresponding acoustic features; The visual and acoustic features are fused using a cross-modal correlation mechanism to reconstruct the target audio.

2. The method for separating specific human audio in a multi-source environment according to claim 1, characterized in that, Suitable for edge computing devices; Correspondingly, the acquisition of noisy frequency signals within the region and the extraction of corresponding acoustic features include: The acoustic features are extracted by processing the noisy frequency signals in the region based on the temporal compression coding strategy. The stride of the convolutional layer is set to an integer greater than or equal to 2 to reduce the temporal dimension length of the feature sequence and adapt to the storage resources of the edge computing device.

3. The method for separating specific human audio in a multi-source environment according to claim 2, characterized in that, The stride of the convolutional layer is set to 4.

4. The method for separating a specific person's audio in a multi-source environment according to claim 3, characterized in that, The acquisition of noisy frequency signals within the region and extraction of corresponding acoustic features includes: The original noisy frequency signal is forcibly resampled to 16kHz; The resampled audio is divided into frames, and the inference duration of each segment is set to 4 seconds. The waveform is normalized so that its values ​​are distributed between [-1, 1].

5. The method for separating specific human audio in a multi-source environment according to claim 4, characterized in that, The step of acquiring the visual image sequence of the target object and extracting the corresponding visual features includes: Identifying the target's lip movements and corresponding lip timing involves the following process: Use a face detection model to locate the lip area; Extract the region of interest (ROI) with the center of the lips as the origin, and fix the size to 96×96 pixels; The video frame rate was uniformly interpolated and adjusted to 25fps; Input visual tensor ,in 100 frames.

6. The method for separating specific human audio in a multi-source environment according to claim 5, characterized in that, The method of fusing the visual features and the acoustic features using a cross-modal correlation mechanism includes: generating an attention mask to suppress noise components in the acoustic features that are inconsistent with the lip timing.

7. The method for separating specific human audio in a multi-source environment according to claim 6, characterized in that, The method of fusing the visual features and the acoustic features using a cross-modal correlation mechanism to reconstruct the target audio includes: Build and train a dedicated multimodal neural network model, including three sub-networks: an audio encoder, a visual encoder, and a cross-modal fusion module; The corresponding hierarchical parameter configurations include: The audio encoder includes: an input layer, which is a one-dimensional convolutional layer, with the number of input channels set to 1, the number of output channels set to 64, the convolutional kernel size set to 8, the stride set to 4, and the padding set to 2; The deep feature extraction network consists of four sequentially connected residual one-dimensional convolutional blocks, used to output the extracted acoustic features. ; The visual encoder includes: a front-end network composed of three-dimensional convolutional layers; a backbone network, which is a deep residual convolutional network with fully connected layers removed; and a temporal alignment module, used to upsample the temporal dimension of the extracted visual features using a nearest neighbor interpolation algorithm to obtain the temporal alignment with the acoustic features. Consistent alignment of visual features across time dimensions ; The cross-modal fusion module is used to generate fused features through a corresponding mechanism: obtaining the query vector Q, key vector K, and value vector V through a mapping layer, wherein... The The above The above These represent the corresponding learnable weight matrices; the attention feature map is calculated using the attention mechanism: ； in, Let K be the dimension of the key vector. This represents the matrix transpose operation; The final fused features are obtained through a gating fusion mechanism. : ； in, Let be the weight matrix of the gating module, sigmoid be the activation function, and ⊙ represent the element-wise multiplication operation of the matrix.

8. The method for separating a specific person's audio in a multi-source environment according to claim 7, characterized in that, The training process of the dedicated neural network model includes: The loss function is constructed based on the scale-invariant signal-to-noise ratio, and the corresponding formula includes: ； Where starget represents the projection component of the target clean speech signal onto the estimated speech signal, and enoise represents the residual noise component in the estimated speech signal; The dedicated neural network model is iteratively trained using a sample set containing noisy frequencies from multiple sound sources and their corresponding clean target audio, until the loss function converges on the validation set.

9. The method for separating specific human audio in a multi-source environment according to claim 8, characterized in that, This applies to financial institutions equipped with edge computing devices; the corresponding area is the counter area, and the target objects are tellers and customers. The method also includes: inputting the reconstructed target audio into a locally deployed automatic speech recognition model for processing.

10. A system for separating the audio of a specific person in a multi-source environment, characterized in that, include: The first module is used to acquire the visual image sequence of the target object and extract the corresponding visual features; The second module is used to acquire noisy frequency signals within the region and extract the corresponding acoustic features; The third module is used to fuse the visual features and the acoustic features using a cross-modal correlation mechanism to reconstruct the target audio.

Images