Cross-modal speech separation method and system based on multi-scale semantic aggregation strategy

Abstract

The application belongs to the field of speech separation, and provides a cross-modal speech separation method and system based on a multi-scale semantic aggregation strategy, acquires a time-domain mixed speech signal and a visual image; encodes the time-domain mixed speech signal, extracts acoustic features; extracts original visual embedding from the visual image, encodes and channel maps the visual features to generate visual features; takes the acoustic features and the visual features as inputs, processes them by using a mask estimation network model, introduces a parallel double-flow cross-modal modeling module based on channel decoupling between the intra-block modeling module and the inter-block modeling module, and uses the cross-modal modeling module to realize bottleneck scale semantic alignment and multi-stage multi-scale collaborative semantic aggregation; decodes the output of the mask estimation network model to obtain target speech. The application generates low-dimensional representation suitable for cross-modal alignment, more effectively reduces the timing redundancy during modeling, thereby greatly reduces the computational complexity while maintaining excellent separation performance.

Description

Technical Field

[0001] This invention belongs to the field of speech separation technology, specifically relating to a cross-modal speech separation method and system based on a multi-scale semantic aggregation strategy. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Multimodal speech separation technology aims to leverage the noise resistance of visual information to assist separation models in extracting the clean speech of the target speaker from mixed speech signals. It is a core front-end module in intelligent conferencing systems, human-computer interaction terminals, and other applications with extremely high computational efficiency requirements. Current mainstream baseline models, such as AV-Sepformer, achieve long sequence modeling through a block-parallel strategy and a dual-path structure. By introducing a cross-modal attention mechanism to fuse visual features, they achieve excellent separation performance and robustness. However, its core Transformer architecture still faces the challenge of computational complexity that grows quadratically with the sequence length. On the one hand, its visual upsampling strategy leads to a large amount of feature duplication and invalid computation; on the other hand, its core Inter-block module maintains a constant temporal resolution when processing global modeling between blocks, ignoring the semantic density differences of features at different modeling stages, resulting in a huge waste of computational power.

[0004] To alleviate this computational bottleneck, researchers have proposed improved architectures based on Transformers, attempting to reduce complexity through low-rank approximation, sparse sampling, kernel functions, or linearized attention. However, these methods are mostly designed for NLP tasks and often struggle to adapt to the modeling characteristics of high-sampling-rate, long-sequence continuous data such as time-domain speech signals. Unlike discrete text, speech signals contain rich, fine-grained time-frequency structure and phase information, and speech separation tasks involve signal reconstruction rather than simple semantic classification.

[0005] Existing general methods such as sparse attention, pruning, or quantization often struggle to retain the fine-grained acoustic features required for separation tasks while significantly reducing model complexity. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a cross-modal speech separation method and system based on a multi-scale semantic aggregation strategy. This invention generates a low-dimensional representation suitable for cross-modal alignment, which more effectively reduces temporal redundancy during modeling, thereby maintaining excellent separation performance while significantly reducing computational complexity.

[0007] According to some embodiments, the present invention adopts the following technical solution: A cross-modal speech separation method based on a multi-scale semantic aggregation strategy includes the following steps: Acquire temporal-domain mixed speech signals and visual images; Encode the time-domain mixed speech signal and extract its acoustic features; The original visual embedding is extracted from the visual image, and the visual features are encoded and mapped to channels to generate visual features; Using acoustic and visual features as input, a mask estimation network model is used for processing. The mask estimation network model is based on a dual-path Transformer architecture. A parallel dual-stream cross-modal modeling module based on channel decoupling is introduced between the intra-block modeling and inter-block modeling modules. The intra-block modeling module is used to extract local semantic maps, the cross-modal modeling module is used to achieve bottleneck-scale semantic alignment, and the inter-block modeling module is used to perform global multi-scale semantic integration to achieve multi-stage, multi-scale collaborative semantic aggregation. The output of the mask estimation network model is decoded to obtain the target speech.

[0008] As an alternative implementation, the process of encoding the temporal mixed speech signal and extracting acoustic features includes: processing the temporal mixed speech signal using an encoder to extract acoustic feature representations, wherein the encoder comprises a one-dimensional convolutional layer and a ReLU activation function; The process of extracting the original visual embedding from a visual image and encoding and mapping the visual features includes: extracting the original visual embedding using a pre-trained lip-reading network, encoding and mapping the original visual embedding using a multi-layer temporal convolutional network, and generating visual features.

[0009] As an alternative implementation, the intra-block modeling module is configured to capture local fine-grained features with overlapping receptive fields using convolutional layers, and then run on the mapped feature space using a standard self-attention mechanism. After the attention mechanism, an inverse aggregation operator consisting of transposed convolutions is introduced, supplemented by linear residual connections, to inject high-frequency details of the original input into the modeling output to compensate for information loss during downsampling.

[0010] As an alternative implementation, the inter-block modeling module is used for global multi-scale semantic integration. The inter-block modeling module performs global modeling, introduces a U-shaped multi-scale semantic aggregation strategy, constructs a hierarchical feature representation from micro to macro, and dynamically adjusts the aggregation scale according to the network depth. In the feature extraction stage and signal reconstruction stage, an aggregation scale smaller than the set value is used to preserve the underlying acoustic details and high-frequency fine-grained information to the greatest extent. In the semantic interaction stage, an aggregation scale larger than the set value is used, and residual connections are introduced to compensate for potential information loss during feature aggregation.

[0011] As an alternative implementation, the cross-modal modeling module is used to achieve bottleneck scale semantic alignment. The modeling process of the cross-modal modeling module includes: decoupling audio features and visual features into two sets of sub-features through channel slicing operations, and feeding them into two parallel branches respectively. The first branch uses a multi-head cross-attention module to process the first set of sub-features, focusing on capturing global semantic alignment across modalities. The second branch uses one-dimensional depthwise separable convolution to process the second set of sub-features. This branch introduces the locality and translation invariance of convolution and focuses on encoding continuous lip changes and short-term acoustic information between adjacent frames. The output features of the two branches are concatenated along the channel dimension and fused with the original input through residual connections.

[0012] As an alternative implementation, the cross-modal modeling module further includes a gated forward propagation network module. The gated forward propagation network module introduces depthwise separable convolution during feature transformation to ensure the continuity of feature transformation. At the same time, it uses gated linear units instead of ReLU activation functions and uses a data-driven gating mechanism to dynamically adjust the transmission of information flow, allowing the network to adaptively suppress irrelevant noise and enhance key features according to the context.

[0013] As an alternative implementation, when performing audiovisual modal sampling, the cross-modal modeling module uses low-resolution video frames as time anchors and actively maps high-resolution audio features to low-resolution visual time scales by learning the semantic correlation of audio features in the time dimension, thereby realizing the extraction of acoustic features in the latent space.

[0014] As an alternative implementation, the process of decoding the output of the mask estimation network model to obtain the target speech includes: decoding using a decoder, which is a one-dimensional transposed convolutional layer with the same kernel size and stride as the encoder; decoding the temporally separated speech signal through audio features and a mask matrix; in the decoding stage, applying the mask matrix to the acoustic features by element-wise multiplication to obtain the separated features, which are then input into the decoder.

[0015] A cross-modal speech separation system based on a multi-scale semantic aggregation strategy includes: The data acquisition module is configured to acquire temporal mixed speech signals and visual images; The encoding module is configured to encode time-domain mixed speech signals and extract acoustic features; extract original visual embeddings from visual images, encode visual features and map them to channels to generate visual features; A multi-stage, multi-scale semantic aggregation module is configured to take acoustic and visual features as inputs and process them using a pre-trained mask estimation network model. The mask estimation network model is based on a dual-path Transformer architecture and introduces a cross-modal modeling module between the intra-block modeling and inter-block modeling modules. The intra-block modeling module is used to extract local semantic maps, the cross-modal modeling module is used to achieve bottleneck scale semantic alignment, and the inter-block modeling module is used to perform global multi-scale semantic integration to achieve multi-stage, multi-scale collaborative semantic aggregation. The decoding module is configured to decode the output of the mask estimation network model to obtain the target speech.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention innovatively introduces a multi-stage, multi-scale collaborative semantic aggregation strategy, constructing an efficient audiovisual-speech separation architecture that breaks through the bottleneck of traditional temporal dual-path models in terms of computational complexity and modeling granularity. Unlike existing sparse attention methods, this invention proposes a multi-stage, multi-scale collaborative temporal semantic aggregation framework tailored to the characteristics of speech signals, performing dynamic granularity matching. Specifically, based on the differentiated information density requirements of different modeling stages (intra-block, cross-modal, and inter-block), three levels of modeling strategies at different scales are designed. This improved strategy not only generates low-dimensional representations suitable for cross-modal alignment but also effectively reduces temporal redundancy during modeling, thereby significantly reducing computational complexity while maintaining excellent separation performance.

[0017] This invention utilizes local convolution as a general semantic aggregation operator, systematically integrating fine-grained to coarse-grained semantic feature aggregation mechanisms into the intra-block local modeling, cross-modal modeling, and inter-block global modeling stages of the dual-path architecture. Through this progressive semantic compression and aggregation, the model not only avoids the computational bottleneck under the original high resolution, but also generates refined high-level semantic information representations, significantly reducing attention complexity while retaining the key feature information required for speech separation tasks.

[0018] This invention designs a parallel dual-stream cross-modal modeling module based on channel decoupling. By decoupling the audiovisual feature space along the channel dimension, it captures global cross-modal semantic associations using multi-head cross-attention branches, while simultaneously extracting local fine-grained temporal features using depthwise separable convolution branches. This design leverages the complementarity of CNNs and Transformers in feature modeling scales, effectively sharing parameter pressure and computational complexity while achieving synergistic optimization of global semantic alignment and local detail enhancement.

[0019] This invention achieves a balance between model separation performance and complexity. Experimental results on benchmark datasets such as VoxCeleb2 show that the proposed method significantly reduces computational complexity (MACs), memory usage, and inference latency while maintaining separation performance on par with the baseline model. Furthermore, as the length of the input audio sequence increases, its efficiency advantage over the baseline model expands significantly and non-linearly, demonstrating its potential for practical deployment.

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1 A flowchart illustrating one embodiment; Figure 2 This is a schematic diagram of a general semantic aggregation operator in one embodiment; Figure 3 This is a schematic diagram of a multi-stage, multi-scale collaborative semantic aggregation strategy in one embodiment; Figure 4 This is a schematic diagram of a multi-stage modeling network architecture in one embodiment, wherein (a) is an intra-block modeling module, (b) is a cross-modal modeling module, and (c) is an inter-block modeling module; Figure 5 This is a schematic diagram of a parallel two-stream cross-modal modeling module according to one embodiment; Figure 6 This is one embodiment of a gated forward propagation network module. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0024] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0025] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0026] Where there is no conflict, the embodiments and features described in this application may be combined with each other.

[0027] Example 1 A cross-modal speech separation method based on a multi-scale semantic aggregation strategy includes the following steps: Acquire temporal-domain mixed speech signals and visual images; Encode the time-domain mixed speech signal and extract its acoustic features; The original visual embedding is extracted from the visual image, and the visual features are encoded and mapped to channels to generate visual features; Using acoustic and visual features as input, a mask estimation network model is used for processing. The mask estimation network model is based on a dual-path Transformer architecture and introduces a cross-modal modeling module between the intra-block modeling and inter-block modeling modules. The intra-block modeling module is used to extract local semantic maps, the cross-modal modeling module is used to achieve bottleneck scale semantic alignment, and the inter-block modeling module is used to perform global multi-scale semantic integration to achieve multi-stage and multi-scale collaborative semantic aggregation. The output of the mask estimation network model is decoded to obtain the target speech.

[0028] The following is a detailed introduction.

[0029] This embodiment utilizes an audiovisual speech separation architecture model to implement the above method. It introduces a stage-multi-scale collaborative temporal semantic aggregation framework to achieve dynamic granularity matching. That is, according to the differentiated information density requirements of different modeling stages (intra-block, cross-modal, and inter-block), three levels of modeling strategies with different scales are designed. This improved strategy not only generates low-dimensional representations suitable for cross-modal alignment, but also effectively reduces temporal redundancy during modeling, thereby maintaining excellent separation performance while significantly reducing computational complexity.

[0030] like Figure 1 As shown, the proposed audiovisual speech separation architecture model follows the encoding-separation-decoding paradigm and consists of an encoder, a masking network, and a decoder. The masking network innovatively introduces a multi-stage, multi-scale collaborative semantic aggregation strategy and a parallel dual-stream cross-modal modeling module.

[0031] The model input is a time-domain mixed speech signal. First, the encoder. right Processing is performed to extract acoustic feature representations. ,in The number of feature channels, The time length is specified. The encoder consists of a one-dimensional convolutional layer (Conv1D) followed by a ReLU activation function.

[0032] (1) In the visual front-end, this embodiment follows the architecture of AV-Sepformer, first using a pre-trained lip-reading network to extract the original visual embeddings. To adapt to the speech separation task, a multi-layer temporal convolutional network (TCN) is used to further encode and channel-map the original visual embeddings to generate visual features. ,in and These represent the feature dimension and the time dimension, respectively.

[0033] Subsequently, and Common Input Mask Estimation Network This network is based on a dual-path Transformer architecture, incorporating intra-block modeling and inter-block modeling. To reduce computational complexity, this embodiment innovatively introduces a multi-stage semantic aggregation and cross-modal modeling module between the Intra and Inter modules, compressing the feature temporal dimension before performing cross-modal modeling and inter-block global interaction. The network ultimately outputs a mask of the target speaker. .

[0034] (2) decoder It is a one-dimensional transposed convolutional layer, where the kernel size and stride are the same as those of the encoder. Similarly, through audio features and mask matrix Decode the time-domain separated speech signal. In the decoding stage, the mask is first processed by element-wise multiplication. Acting on acoustic features The separated features are obtained and then input into the decoder. The decoder consists of a one-dimensional transposed convolutional layer (TransConv1D), whose kernel size and stride configuration are the inverse operations of the encoder, ultimately reconstructing the time-domain separated speech signal. .

[0035] (3) in, This indicates the transpose convolution operation. This represents the Hadamard product by elements.

[0036] To achieve efficient computation across the entire chain, this embodiment proposes a multi-scale temporal feature aggregation framework and defines a general semantic feature aggregation operator. ,like Figure 2 As shown. For the input feature sequence ,in For the number of channels, Given the initial sequence length, this operator aggregates high-frequency fine-grained features into compact coarse-grained semantic units through parameterized convolutional transformations: (4) in The kernel size is [size]. For aggregation factor, This is the activation function. This operator not only reduces the dimensionality of the sequence length but also performs semantic aggregation of information. Based on this unified paradigm, three specific modeling modules are designed to meet the semantic information granularity requirements at different stages, such as... Figure 3 As shown.

[0037] In-block modeling based on local scale semantic mapping While the standard Transformer excels at capturing long-range dependencies, it suffers from two limitations when performing intra-block modeling: first, the self-attention mechanism gives equal attention to features across all temporal sequences, neglecting significant local correlations in the speech signal; second, the temporal redundancy caused by the high sampling rate results in enormous computational overhead for attention at full resolution. To address these limitations, this embodiment proposes a locally enhanced feature aggregation architecture.

[0038] Convolutional pre-aggregation: 1D convolutions with a stride of 2 are used to preferentially capture local fine-grained features with overlapping receptive fields. The aggregation factor is then... Setting it to 2 not only reduces the sequence length to the initial value, but also... Furthermore, it provides pre-selected potential features with local correlations for subsequent attention mechanisms, enhancing the model's ability to capture subtle acoustic details.

[0039] Aggregate domain attention: The standard self-attention mechanism operates in the mapped feature space, and the reduction in sequence length significantly reduces the computational cost.

[0040] Feature recovery and residuals: Since speech separation is a generative task, it is necessary to maintain high temporal resolution of the waveform. Therefore, this embodiment introduces an inverse aggregation operator composed of transposed convolutions after the attention mechanism. Furthermore, by employing linear residual connections, high-frequency details of the original input are injected into the modeling output to compensate for information loss during the downsampling process.

[0041] Figure 4(a) illustrates the intra-block modeling module for local feature enhancement. The specific modeling process includes: (5) (6) (7) Cross-modal modeling based on bottleneck-scale semantic alignment: To address the inherent sampling rate mismatch problem in audiovisual modalities, existing methods (such as AV-Sepformer) typically employ visual upsampling strategies. However, this passive replication of visual features not only fails to introduce additional information but also increases the computational burden of cross-attention. To solve this problem, this embodiment proposes an inverse resolution alignment strategy, namely "audio-adaptive-visual." This strategy uses low-resolution video frames as temporal anchors and actively maps high-resolution audio features to a compact low-resolution visual temporal scale by learning the semantic relevance of audio features in the temporal dimension, thus achieving acoustic feature extraction in the latent space.

[0042] like Figure 4 As shown in (b), it specifically includes: audiovisual alignment: This embodiment uses depthwise separable convolution as the aggregation function. This design utilizes the local inductive bias of CNNs to achieve audiovisual alignment with a very low number of parameters. Each adjacent audio frame is aggregated into a single representative feature with high semantic density, thereby achieving alignment with the video frame in the latent space. In AV-Sepformer's 2D positional coding, each video frame is aligned with the center position of its corresponding audio block, and this standard is maintained in the alignment strategy.

[0043] Cross-modal modeling and reconstruction: The aggregated audio key / value pairs interact with the visual query in a low-dimensional and efficient manner. Subsequently, the fused features are projected back to the original resolution through an upsampling layer, and residual connections are introduced to add the multimodal fused features to the initial audio features. This not only ensures the integrity of the waveform reconstruction but also effectively alleviates the vanishing gradient problem, ensuring the training stability of deep networks.

[0044] Visual features contain both time-invariant static global information (such as speaker identity, gender, and facial contours) and dynamic temporal information, namely, continuous changes in lip shape between adjacent video frames. Both types of information are indispensable for extracting target speech from mixed signals. However, the AV-Sepformer architecture employs a full-channel feature interaction mode, applying a uniform attention mechanism across the entire feature dimension. This may cause the model to overemphasize certain keyframes with significant identity features, while neglecting the temporal dependencies between adjacent frames, resulting in inefficient modeling of local acoustic details.

[0045] To overcome the above limitations, this embodiment proposes a parallel two-stream cross-modal modeling architecture based on channel decoupling. For example... Figure 5 As shown, the architecture follows the "divide and conquer" design concept, decoupling the audiovisual feature space into two independent sub-feature spaces in the channel dimension, which are respectively processed in parallel by the attention branch of long-range modeling and the convolution branch of local modeling.

[0046] The specific modeling process is as follows: Assume the input audiovisual feature sequence is: ,in It is the number of block features. It refers to the number of channels. First, it is decoupled into two sets of sub-features through channel slicing: , Among them, satisfying The data is fed into two parallel branches. The first branch uses a multi-head cross-attention (MHCA) module for processing. This method focuses on capturing global semantic alignment across modalities. Due to the halving of the number of channels, the computational complexity of this process is significantly reduced. The computational process is described as follows: (8) The second branch utilizes one-dimensional depthwise separable convolution. This branch road ( This approach introduces the locality and translation invariance inherent in convolution, focusing on encoding continuous lip changes and short-term acoustic information between adjacent frames, effectively compensating for the shortcomings of attention mechanisms in micro-temporal modeling. The process is represented as follows: (9) Subsequently, the output characteristics of the two branches and The input is concatenated along the channel dimension and fused with the original input via residual connections. To further enhance the model's ability to capture local temporal correlations, a gated feed-forward network (GFFN) is introduced after the two-stream module to replace the standard feed-forward neural network (FFN), forming a feature representation that combines global semantics with local details.

[0047] (10) in, This indicates feature splicing.

[0048] The specific structure of the gated forward propagation network module is as follows: Figure 6As shown. The Gated Forward Propagation Network (GFFN) module aims to overcome the shortcomings of the standard FFN's point-to-point processing method, which ignores the temporal context. GFFN explicitly introduces a depthwise separable convolution with a kernel size of 3 during feature transformation, ensuring the continuity of feature transformation. Simultaneously, it uses a Gated Linear Unit (GLU) instead of the traditional ReLU activation function, leveraging a data-driven gating mechanism to dynamically adjust the information flow, allowing the network to adaptively suppress irrelevant noise and enhance key features based on the context. Its calculation formula is defined as follows: (11) in, and It is a learnable weight matrix. It is the Sigmoid activation function.

[0049] Inter-block modeling based on global multi-scale semantic integration like Figure 4 As shown in (c), in the dual-path architecture, the Intra module focuses on short-term microstructures, extracting local acoustic details. The Inter module aims to capture long-term macro-dependencies (such as speaker identity and intonation continuity). After deep interaction in the Intra stage, the intra-block features exhibit high semantic saturation, and the attention distribution across adjacent temporal dimensions tends to be consistent. AV-Sepformer continues to use full-resolution inter-block modeling, i.e., using fine-grained features to complete coarse-grained global modeling goals. This modeling process and output with different semantic granularities not only introduces a large amount of redundant attention computation but may also cause the model to be interfered with by irrelevant high-frequency noise when processing long-distance dependencies. This leads to serious computational waste and high-frequency noise interference.

[0050] Based on the above analysis, this embodiment introduces a U-shaped multi-scale semantic aggregation strategy in the inter-block modeling stage to construct a hierarchical feature representation from micro to macro levels. The aggregation scale is dynamically adjusted according to the network depth. , among which is Temporal length of block features This refers to the aggregation factor. In the shallow layers (feature extraction stage) and deep layers (signal reconstruction stage), a lower aggregation scale is used to preserve the underlying acoustic details and high-frequency fine-grained information to the greatest extent. In the intermediate layers (semantic interaction stage), a higher aggregation scale is used. This process can be viewed as a mapping from frame-level feature space to fragment-level semantic space, forcing the model to perform global interactions on a highly abstract fragment-level semantic space. Furthermore, residual connections are introduced in the high-aggregation-scale intermediate layers to compensate for potential information loss during feature aggregation. Specifically, this is achieved by designing a stride factor. Greater than 1 and kernel size is Depth-separable convolutions are used for downsampling. In the 8-layer Inter module, layers 1, 2, 7, and 8 are configured for fine-grained ( Layers 3 through 6 are configured with a coarse grain (), while layers 3 through 6 are configured with a coarse grain (). This design not only significantly reduces computational complexity but also enables the model to focus on long-range semantic features, achieving an optimal balance between performance and efficiency.

[0051] This embodiment demonstrates the superiority of the proposed method by quantitatively analyzing the complexity changes of core computational bottlenecks at different modeling stages within the entire modular architecture. In the Transformer structure, computational complexity is primarily determined by the feature dimension and sequence length. Assume the input feature tensor is... Where S is the number of blocks, K is the length of each block, and D is the number of channels, i.e., the feature dimension. This embodiment mainly analyzes the core computational complexity in a single Transformer layer, as shown in Table 1.

[0052] Table 1. Comparison of core computational costs between the original architecture and the proposed method.

[0053] Learning Objectives The proposed model architecture is trained end-to-end, and the objective function is the scale-invariant signal-to-noise ratio (SI-SNR), as shown in Equation (12): (12) (13) in, Used to evaluate target separation signal With pure signal Separation quality between them. Optimal scaling factor to eliminate amplitude differences. It is given by formula (13).

[0054] Example 2 A cross-modal speech separation system based on a multi-scale semantic aggregation strategy includes: The data acquisition module is configured to acquire temporal mixed speech signals and visual images; The encoding module is configured to encode time-domain mixed speech signals and extract acoustic features; extract original visual embeddings from visual images, encode visual features and map them to channels to generate visual features; A multi-stage, multi-scale semantic aggregation module is configured to take acoustic and visual features as inputs and process them using a mask estimation network model. The mask estimation network model is based on a dual-path Transformer architecture and introduces a cross-modal modeling module between the intra-block modeling and inter-block modeling modules. The intra-block modeling module is used to extract local semantic maps, the cross-modal modeling module is used to achieve bottleneck-scale semantic alignment, and the inter-block modeling module is used to perform global multi-scale semantic integration to achieve multi-stage, multi-scale collaborative semantic aggregation. The decoding module is configured to decode the output of the mask estimation network model to obtain the target speech.

[0055] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can be implemented in one or more computer-usable storage media (including, but not limited to, disk storage, etc.) containing computer-usable program code. CD - ROM It takes the form of a computer program product implemented on (such as optical memory, etc.).

[0056] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0059] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art without creative effort within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cross-modal speech separation method based on a multi-scale semantic aggregation strategy, characterized in that, Includes the following steps: Acquire temporal-domain mixed speech signals and visual images; Encode the time-domain mixed speech signal and extract its acoustic features; The original visual embedding is extracted from the visual image, and the visual features are encoded and mapped to channels to generate visual features; Using acoustic and visual features as input, a mask estimation network model is used for processing. The mask estimation network model is based on a dual-path Transformer architecture. A cross-modal modeling module based on channel decoupling is introduced between the intra-block modeling and inter-block modeling modules. The intra-block modeling module is used to extract local semantic maps, the cross-modal modeling module is used to achieve bottleneck-scale semantic alignment, and the inter-block modeling module is used to perform global multi-scale semantic integration to achieve multi-stage and multi-scale collaborative semantic aggregation. The output of the mask estimation network model is decoded to obtain the target speech. The intra-block modeling module is configured to capture local fine-grained features with overlapping receptive fields using convolutional layers, and then run on the mapped feature space using a standard self-attention mechanism. After the attention mechanism, an inverse aggregation operator composed of transposed convolutions is introduced, supplemented by linear residual connections, to inject high-frequency details of the original input into the modeling output to compensate for information loss during downsampling. The inter-block modeling module is used for global multi-scale semantic integration. The inter-block modeling module performs global modeling, introduces a U-shaped multi-scale semantic aggregation strategy, constructs a hierarchical feature representation from micro to macro, and dynamically adjusts the aggregation scale according to the network depth. In the feature extraction stage and signal reconstruction stage, an aggregation scale smaller than the set value is used to preserve the underlying acoustic details and high-frequency fine-grained information to the greatest extent. In the semantic interaction stage, an aggregation scale larger than the set value is used, and residual connections are introduced to compensate for potential information loss in the feature aggregation process. The cross-modal modeling module is used to achieve bottleneck scale semantic alignment. The modeling process of the cross-modal modeling module includes: decoupling audio features and visual features into two sets of sub-features through channel slicing operation, and feeding them into two parallel branches respectively. The first branch uses a multi-head cross-attention module to process the first set of sub-features, focusing on capturing global semantic alignment across modalities. The second branch uses one-dimensional depthwise separable convolution to process the second set of sub-features. This branch introduces the locality and translation invariance of convolution and focuses on encoding continuous lip changes and short-term acoustic information between adjacent frames. The output features of the two branches are concatenated along the channel dimension and fused with the original input through residual connections.

2. The cross-modal speech separation method based on a multi-scale semantic aggregation strategy as described in claim 1, characterized in that, The process of encoding a temporal mixed speech signal and extracting acoustic features includes: processing the temporal mixed speech signal using an encoder to extract acoustic feature representations, wherein the encoder consists of a one-dimensional convolutional layer and a ReLU activation function; The process of extracting the original visual embedding from a visual image and encoding and mapping the visual features includes: extracting the original visual embedding using a pre-trained lip-reading network, encoding and mapping the original visual embedding using a multi-layer temporal convolutional network, and generating visual features.

3. The cross-modal speech separation method based on a multi-scale semantic aggregation strategy as described in claim 1, characterized in that, The cross-modal modeling module also includes a gated forward propagation network module. The gated forward propagation network module introduces depthwise separable convolution during feature transformation to ensure the continuity of feature transformation. At the same time, it uses gated linear units to replace the ReLU activation function and uses a data-driven gating mechanism to dynamically adjust the transmission of information flow, allowing the network to adaptively suppress irrelevant noise and enhance key features according to the context.

4. The cross-modal speech separation method based on a multi-scale semantic aggregation strategy as described in claim 3, characterized in that, The depthwise separable convolution, as an aggregation function, utilizes the local inductive bias of the convolutional neural network to aggregate K adjacent audio frames into a single representative feature with high semantic density using a parameter amount less than a set value. This achieves strict alignment with video frames in the latent space, with each video frame aligned with the center position of its corresponding audio block.

5. The cross-modal speech separation method based on a multi-scale semantic aggregation strategy as described in claim 1, characterized in that, When performing audiovisual modal sampling, the cross-modal modeling module uses low-resolution video frames as time anchors and actively maps high-resolution audio features to low-resolution visual time scales by learning the semantic correlation of audio features in the time dimension, thereby realizing the extraction of acoustic features in the latent space.

6. The cross-modal speech separation method based on a multi-scale semantic aggregation strategy as described in claim 1, characterized in that, The process of decoding the output of the mask estimation network model to obtain the target speech includes: decoding using a decoder, which is a one-dimensional transposed convolutional layer with the same kernel size and stride as the encoder; decoding the temporally separated speech signal through audio features and the mask matrix; in the decoding stage, the mask matrix is ​​applied to the acoustic features by element-wise multiplication to obtain the separated features, which are then input into the decoder.

7. A cross-modal speech separation system based on a multi-scale semantic aggregation strategy, characterized in that, include: The data acquisition module is configured to acquire temporal mixed speech signals and visual images; The encoding module is configured to encode time-domain mixed speech signals and extract acoustic features; The original visual embedding is extracted from the visual image, and the visual features are encoded and mapped to channels to generate visual features; A multi-stage, multi-scale semantic aggregation module is configured to take acoustic and visual features as inputs and process them using a mask estimation network model. The mask estimation network model is based on a dual-path Transformer architecture and introduces a cross-modal modeling module between the intra-block modeling and inter-block modeling modules. The intra-block modeling module is used to extract local semantic maps, the cross-modal modeling module is used to achieve bottleneck-scale semantic alignment, and the inter-block modeling module is used to perform global multi-scale semantic integration to achieve multi-stage, multi-scale collaborative semantic aggregation. The decoding module is configured to decode the output of the mask estimation network model to obtain the target speech; The intra-block modeling module is configured to capture local fine-grained features with overlapping receptive fields using convolutional layers, and then run on the mapped feature space using a standard self-attention mechanism. After the attention mechanism, an inverse aggregation operator composed of transposed convolutions is introduced, supplemented by linear residual connections, to inject high-frequency details of the original input into the modeling output to compensate for information loss during downsampling. The inter-block modeling module is used for global multi-scale semantic integration. The inter-block modeling module performs global modeling, introduces a U-shaped multi-scale semantic aggregation strategy, constructs a hierarchical feature representation from micro to macro, and dynamically adjusts the aggregation scale according to the network depth. In the feature extraction stage and signal reconstruction stage, an aggregation scale smaller than the set value is used to preserve the underlying acoustic details and high-frequency fine-grained information to the greatest extent. In the semantic interaction stage, an aggregation scale larger than the set value is used, and residual connections are introduced to compensate for potential information loss in the feature aggregation process. The cross-modal modeling module is used to achieve bottleneck scale semantic alignment. The modeling process of the cross-modal modeling module includes: decoupling audio features and visual features into two sets of sub-features through channel slicing operation, and feeding them into two parallel branches respectively. The first branch uses a multi-head cross-attention module to process the first set of sub-features, focusing on capturing global semantic alignment across modalities. The second branch uses one-dimensional depthwise separable convolution to process the second set of sub-features. This branch introduces the locality and translation invariance of convolution and focuses on encoding continuous lip changes and short-term acoustic information between adjacent frames. The output features of the two branches are concatenated along the channel dimension and fused with the original input through residual connections.

Images